Copy page

Version: Next

On Your Environment

Install the Agent Manager on an existing Kubernetes cluster — AWS EKS, Google GKE, Azure AKS, or any distribution with LoadBalancer support.

Just want it running locally?

The Quick Start Guide installs everything in a single command using a dev container with k3d. Use this page when you need to install on a managed cluster.

Stronger agent isolation tiers

Agents run sandboxed under the standard runc runtime by default. Agent Manager also supports stronger per-environment isolation tiers — gVisor (userspace kernel) and Kata Containers (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the gVisor and Kata Containers setup guides.

What You Will Get

Agent Manager is a two-layer system installed in two phases:

Phase 1 — OpenChoreo (base layer): OpenChoreo is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a Control Plane for API and configuration, a Data Plane for running workloads and gateways, a Workflow Plane for builds and CI pipelines, and an Observability Plane for traces, logs, and metrics via OpenSearch.
Phase 2 — Agent Manager : The AI agent management platform installed on top of OpenChoreo. It includes the Console (web UI), AMP API (backend), AI Gateway, PostgreSQL (database), Secrets Extension (OpenBao for runtime secret injection), Agent Manager Observer (traces, logs, and metrics), and Evaluation Engine (automated agent evaluations).

This guide installs both layers on your existing Kubernetes cluster, production-first: a real domain, trusted TLS certificates issued automatically by cert-manager, and every management surface served through the three OpenChoreo plane gateways — the only LoadBalancers the platform needs.

No domain yet?

The main flow assumes you control a DNS zone. If you are evaluating on a cluster without one, follow the nip.io appendix instead — same steps, with hostnames derived from LoadBalancer IPs and self-signed certificates.

Prerequisites

Cluster Requirements

Requirement	Minimum
Kubernetes version	1.32+
Nodes	3
CPU per node	4 cores
RAM per node	8 GB
LoadBalancer support	Required
Default StorageClass	Required
Node kernel (build nodes)	Linux 6.3+ for user-namespaced agent builds (idmapped-mount support). On older kernels (Ubuntu 22.04's 5.15, Amazon Linux 2's 5.10) set `buildWorkflows.userNamespaces=false` on the Platform Resources chart — see Troubleshooting

Supported Providers

Amazon Web Services (EKS)
Google Cloud Platform (GKE)
Microsoft Azure (AKS)
Any Kubernetes distribution with LoadBalancer support

Required Tools

Tool	Version	Purpose
kubectl	v1.32+	Kubernetes CLI
Helm	v3.12+	Kubernetes package manager
curl / dig	—	DNS resolution of LoadBalancer hostnames

kubectl version --client && helm version

Permissions

You need sufficient privileges to:

Create namespaces, deploy Helm charts
Create LoadBalancer services (only the OpenChoreo plane gateways need them — the Agent Manager services are ClusterIP and are served through those gateways)
Manage cert-manager Issuers and Certificates
Create CRDs and ClusterRoles

Plan Your Deployment

Three decisions shape the installation. Make them now — everything below is driven by the variables you export here.

1. Hostnames

Pick a base domain you control (a delegated subdomain like amp.yourdomain.com works well). Every management surface is a hostname on one of the three plane gateway LoadBalancers — after the platform installs, you publish DNS records against just those three targets:

Hostname	Serves	DNS record →
`console.<base>`	Agent Manager Console	control-plane gateway LB
`api-amp.<base>`	Agent Manager API (browser, `amctl`, MCP)	control-plane gateway LB
`thunder.<base>`	Thunder OAuth login (+ per-environment `<org>-<env>.thunder.<base>`)	control-plane gateway LB
`cp.<base>`	Gateway control plane (external AI gateways, optional)	control-plane gateway LB
`api.<base>`	OpenChoreo API	control-plane gateway LB
`traces.<base>`	Agent Manager Observer	observability-plane gateway LB
`*.agents.<base>`	Deployed-agent invocation endpoints (wildcard)	data-plane gateway LB

Concretely: console/api-amp/thunder/cp/api all point at the control-plane gateway (five records, or a single *.<base> wildcard), traces points at the observability-plane gateway, and *.agents.<base> plus the agents.<base> apex point at the data-plane gateway. Step 10 lists the exact records once the LoadBalancer addresses exist.

export VERSION="0.0.0-dev"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"

# Your base domain — everything below derives from it
export BASE_DOMAIN="amp.yourdomain.com"

export CONSOLE_PUBLIC_HOST="console.${BASE_DOMAIN}"
export API_PUBLIC_HOST="api-amp.${BASE_DOMAIN}"
export THUNDER_PUBLIC_HOST="thunder.${BASE_DOMAIN}"
export CP_GW_PUBLIC_HOST="cp.${BASE_DOMAIN}"
export OBS_API_PUBLIC_HOST="traces.${BASE_DOMAIN}"
export AGENTS_DOMAIN="agents.${BASE_DOMAIN}"

export CONSOLE_PUBLIC_URL="https://${CONSOLE_PUBLIC_HOST}"
export API_PUBLIC_URL="https://${API_PUBLIC_HOST}"
export THUNDER_PUBLIC_URL="https://${THUNDER_PUBLIC_HOST}"
export OBS_API_PUBLIC_URL="https://${OBS_API_PUBLIC_HOST}"

# In-cluster URL backend services use for Thunder JWKS/token calls
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"

# Trace-export endpoint shown to agent developers (see the OTLP note below)
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"

Why api-amp and not api?

The OpenChoreo API claims api.<base> on the same gateway, and the certificates below are single-level wildcards (*.<base>), so every hostname must sit directly under the base domain.

2. TLS certificates

The main flow uses cert-manager with Let's Encrypt (DNS-01): publicly trusted certificates, automatic renewal, and wildcard support — you only need API credentials for the DNS provider hosting your zone. Step 3 also shows a corporate-CA alternative. Certificate issuance needs only the DNS zone, so it works before the DNS records for the gateways exist.

3. Dependencies

The default install runs PostgreSQL, OpenBao, and OpenSearch in-cluster with development settings. Each install step below carries a Production callout with the values to swap in a managed database, a hardened secrets backend, and persistent observability storage. Decide up front which you will use — moving PostgreSQL data or Thunder's identity database later is disruptive.

OTLP trace ingest

INSTRUMENTATION_URL is the trace-export endpoint shown to agent developers; the traffic rides the data-plane gateway's /otel route. Agents running inside the cluster can use the default route hostname http://default-default.gateway.localhost:19080/otel (resolvable in-cluster). For agents running outside the cluster, install the API Platform Gateway extension with a public hostname whose DNS points at the data-plane gateway LoadBalancer — --set gateway.vhost=https://otel.${BASE_DOMAIN} and --set gateway.hostname=otel.${BASE_DOMAIN} — and set INSTRUMENTATION_URL=https://otel.${BASE_DOMAIN}/otel.

External AI gateways

AI gateways running outside this cluster connect through the control-plane gateway at https://${CP_GW_PUBLIC_HOST}. Phase 2 passes this hostname to the Agent Manager install; nothing else is needed beyond the DNS record. See Register an AI Gateway.

Phase 1: OpenChoreo Platform

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

Control Plane — API server and configuration management for the platform
Data Plane — runs deployed workloads and API gateways
Workflow Plane — builds and CI pipelines for agent deployments
Observability Plane — trace, log, and metrics collection via OpenSearch

This phase also installs Thunder (the identity provider) as a prerequisite, since the Control Plane and Observability Plane require Thunder's OIDC endpoints for JWT validation. Estimated time: ~20-30 minutes (varies by cluster and network).

Step 1: Install Cluster Prerequisites

Gateway API CRDs (v1.4.1) — standard Kubernetes resources for managing network gateways and routing:

kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

info

The --force-conflicts flag is needed if your cluster already has Gateway API CRDs managed by another controller (e.g., Traefik on k3s/Rancher Desktop).

cert-manager (v1.19.2) — automates TLS certificate issuance and renewal:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --set startupapicheck.timeout=5m \
  --wait --timeout 360s

External Secrets Operator (v1.3.2) — syncs secrets from external stores (like OpenBao) into Kubernetes:

helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s

kgateway (v2.2.1) — the network gateway for OpenChoreo planes:

helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true

Rancher Desktop / k3s users

k3s ships with Traefik which binds to host ports 80/443 and conflicts with OpenChoreo's kgateway. Remove Traefik before proceeding:

helm uninstall traefik -n kube-system
helm uninstall traefik-crd -n kube-system

After removing Traefik, re-apply the Gateway API CRDs (Traefik's CRD chart may have removed them):

kubectl apply --server-side --force-conflicts \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

Step 2: Setup Secrets Store (OpenBao)

OpenBao provides secret management for the Workflow Plane and deployed agents:

helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-openbao.yaml \
  --timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s

Configure the External Secrets ClusterSecretStore:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF

Production

The values file above runs OpenBao in dev mode (in-memory backend, auto-unseal, well-known root token) — secrets are lost on pod restart. For production, install OpenBao without the dev-mode values: set server.dev.enabled=false, enable server.dataStorage persistence (or an external storage backend), configure auto-unseal, and create the Kubernetes auth role (openchoreo-secret-writer-role) plus the KV v2 mount at secret/ that the ClusterSecretStore below expects.

Step 3: Setup TLS Issuer

Every gateway certificate below references one ClusterIssuer named openchoreo-ca. Create it from your chosen certificate authority — the rest of the guide is identical regardless of which tab you pick.

Let's Encrypt (DNS-01)
Corporate CA
Self-signed (evaluation only)

DNS-01 validation issues publicly trusted certificates (including the wildcards this guide needs) by writing a TXT record to your DNS zone — the CA never connects to your cluster. Create a credentials Secret for your DNS provider, then the issuer. The example below uses AWS Route 53; cert-manager supports Cloudflare, Google Cloud DNS, Azure DNS, and others with the same structure.

kubectl create secret generic route53-credentials \
  --namespace cert-manager \
  --from-literal=secret-access-key='<AWS_SECRET_ACCESS_KEY>'

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@yourdomain.com
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - dns01:
          route53:
            region: us-east-1
            accessKeyID: <AWS_ACCESS_KEY_ID>
            secretAccessKeySecretRef:
              name: route53-credentials
              key: secret-access-key
        selector:
          dnsZones:
            - "${BASE_DOMAIN}"
EOF

tip

Test with the Let's Encrypt staging server (https://acme-staging-v02.api.letsencrypt.org/directory) first if you expect to iterate — the production server rate-limits duplicate certificates to 5 per week.

If your organization runs an internal PKI, store its intermediate CA certificate and key as a Secret and create a CA issuer. Clients inside your network trust the certificates via your existing root-CA distribution.

kubectl create secret tls corporate-ca-secret \
  --namespace cert-manager \
  --cert=/path/to/intermediate-ca.crt \
  --key=/path/to/intermediate-ca.key

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  ca:
    secretName: corporate-ca-secret
EOF

For evaluation without a domain, the nip.io appendix creates a self-signed chain under the same openchoreo-ca name. Browsers will warn on every hostname and backend components need TLS-verification overrides — do not use this beyond evaluation.

Step 4: Install Thunder Extension (Identity Provider)

Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder. The browser reaches it at https://${THUNDER_PUBLIC_HOST} through the control-plane gateway.

helm install amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \
  --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \
  --set thunder.configuration.gateClient.scheme=https \
  --set thunder.configuration.gateClient.port=443 \
  --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \
  --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/instance=amp-thunder-extension \
  -n ${THUNDER_NS} --timeout=300s

warning

Thunder persists its configuration (including the issuer URL and the console client's OAuth redirect URIs) in a database on first boot. If you need to change THUNDER_PUBLIC_URL or CONSOLE_PUBLIC_URL after installation, you must uninstall the chart, delete its PVC, and reinstall — a helm upgrade alone will not change the issuer in issued tokens or the registered redirect URIs. This is why the hostnames are fixed in Plan Your Deployment before anything installs.

Verify

kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"${THUNDER_PUBLIC_URL}"  (must match your THUNDER_PUBLIC_URL value)

Step 5: Install OpenChoreo Control Plane

Because the hostnames are fixed up front, the Control Plane installs once with its final configuration. Create the gateway's wildcard certificate first (issuance needs only the DNS zone, not the DNS records):

kubectl create namespace openchoreo-control-plane --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cp-gateway-tls
  namespace: openchoreo-control-plane
spec:
  secretName: cp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/cp-gateway-tls \
  -n openchoreo-control-plane --timeout=300s

Install the Control Plane with hostnames, TLS, and Thunder OIDC:

helm upgrade --install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.1.1 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --values - <<EOF
openchoreoApi:
  config:
    server:
      publicUrl: "https://api.${BASE_DOMAIN}"
  http:
    hostnames:
      - "api.${BASE_DOMAIN}"
backstage:
  enabled: false
  baseUrl: ""
  http:
    hostnames:
      - ""
security:
  oidc:
    issuer: "${THUNDER_PUBLIC_URL}"
    wellKnownEndpoint: "${THUNDER_INTERNAL_URL}/.well-known/openid-configuration"
    jwksUrl: "${THUNDER_INTERNAL_URL}/oauth2/jwks"
    authorizationUrl: "${THUNDER_PUBLIC_URL}/oauth2/authorize"
    tokenUrl: "${THUNDER_INTERNAL_URL}/oauth2/token"
gateway:
  tls:
    enabled: true
    hostname: "*.${BASE_DOMAIN}"
    certificateRefs:
      - name: cp-gateway-tls
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=300s

Webhook race condition

If the install fails with no endpoints available for service "controller-manager-webhook-service", this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the helm upgrade --install command.

kubectl wait --for=condition=Available deployment --all \
  -n openchoreo-control-plane --timeout=300s

EKS Users

EKS LoadBalancers return a hostname instead of an IP — resolve it with dig when you publish DNS records later. For internet-facing access, annotate the gateway Service with service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing.

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)

Thunder 0.45+ puts the client name in the client_id claim (its sub is an opaque UUID), but OpenChoreo 1.1.1 extracts service-account entitlements from sub, and the chart does not expose the setting. Without this patch, every service-to-service call is silently unauthorized — the API returns 200 with empty lists and 403s while all pods look healthy, and the gateway bootstrap later fails with Environment 'default' not found.

patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s

for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
  claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
  if [ "$claim" = "sub" ]; then
    kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge       -p '{"spec":{"entitlement":{"claim":"client_id"}}}'
  fi
done

note

Re-apply this patch after any helm upgrade of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's observer-auth-config in Step 8.

What the configuration does

Backstage disabled (AMP provides its own console)
OIDC issuer set to THUNDER_PUBLIC_URL (matches the iss claim in Thunder-issued JWTs)
OIDC JWKS URL points to Thunder's in-cluster service (avoids external DNS dependency, and no TLS-verification overrides are needed since the in-cluster call is plain HTTP)
OpenChoreo API at api.${BASE_DOMAIN}
TLS enabled on the gateway with the wildcard certificate — Thunder, the Console, the API, and the gateway control plane all serve HTTPS under it

Step 6: Setup Data Plane

Copy the cluster-gateway CA certificate:

kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

Create the deployed-agents wildcard certificate, then install the Data Plane with TLS in one pass:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dp-gateway-tls
  namespace: openchoreo-data-plane
spec:
  secretName: dp-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${AGENTS_DOMAIN}"
    - "${AGENTS_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/dp-gateway-tls \
  -n openchoreo-data-plane --timeout=300s

helm install openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.1.1 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${AGENTS_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=dp-gateway-tls" \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-dp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s

CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "${AGENTS_DOMAIN}"
          listenerName: http
          port: 80
        https:
          host: "${AGENTS_DOMAIN}"
          listenerName: https
          port: 443
  secretStoreRef:
    name: default
EOF

Step 7: Setup Workflow Plane

Copy the cluster-gateway CA certificate:

kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

Container Registry

The Workflow Plane needs a container registry to store built agent images. The registry endpoint is configured in Phase 2 Step 3 (Platform Resources) via the global.registry.endpoint or global.baseDomain Helm values. For local development, deploy an in-cluster docker-registry in this namespace — see the k3d guide for an example.

Install the Workflow Plane:

helm install openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.1.1 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --set clusterAgent.tls.generateCerts=true \
  --timeout 600s

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s

BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
  secretStoreRef:
    name: default
EOF

Step 8: Setup Observability Plane

Copy the cluster-gateway CA certificate:

kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

TLS_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.crt}' | base64 -d)
TLS_KEY=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.tls\.key}' | base64 -d)
kubectl create secret generic cluster-gateway-ca \
  --from-literal=tls.crt="$TLS_CRT" \
  --from-literal=tls.key="$TLS_KEY" \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

Create the ExternalSecrets for OpenSearch and Observer credentials:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF

Wait for the ExternalSecrets to sync:

kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/oc-collector-configmap.yaml \
  -n openchoreo-observability-plane

Create the gateway certificate, then install the Observability Plane:

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: obs-gateway-tls
  namespace: openchoreo-observability-plane
spec:
  secretName: obs-gateway-tls
  issuerRef:
    name: openchoreo-ca
    kind: ClusterIssuer
  dnsNames:
    - "*.${BASE_DOMAIN}"
    - "${BASE_DOMAIN}"
  privateKey:
    rotationPolicy: Always
EOF

kubectl wait --for=condition=Ready certificate/obs-gateway-tls \
  -n openchoreo-observability-plane --timeout=300s

helm install openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.1.1 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --set gateway.tls.enabled=true \
  --set "gateway.tls.hostname=*.${BASE_DOMAIN}" \
  --set "gateway.tls.certificateRefs[0].name=obs-gateway-tls" \
  --set clusterAgent.tls.generateCerts=true \
  --set observer.controlPlaneApiUrl="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --set observer.extraEnv.AUTH_SERVER_BASE_URL="${THUNDER_PUBLIC_URL}" \
  --set security.oidc.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set security.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/single-cluster/values-op.yaml \
  --timeout 25m

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=900s

for sts in $(kubectl get statefulset -n openchoreo-observability-plane -o name 2>/dev/null); do
  kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s
done

With Thunder ≥ 0.45, apply the same entitlement-claim patch from Step 5 to the observer (without it, build-log queries return 403 Access denied):

patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml   | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s

Install observability modules (logs, metrics, tracing):

# Logs module
helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials" \
  --timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.6.1 \
  --timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m

Production

OpenSearch stores all traces, logs, and metrics. The default modules run it with ephemeral storage — set persistence before real use (--set opensearch.persistence.enabled=true --set opensearch.persistence.size=100Gi on the logs module that deploys OpenSearch, sized to your retention needs), or point the observability stack at a managed OpenSearch domain if your platform team operates one.

OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo-observability-plane.svc.cluster.local:8080
EOF

# Link Data Plane to Observability
kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability
kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

Step 9: Verify OpenChoreo Installation

Before proceeding to Phase 2, confirm all planes are running:

echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default

All pods should be in Running or Completed state.

Step 10: Publish DNS Records

The three plane gateways have LoadBalancers now — read their addresses and create the DNS records planned in Plan Your Deployment:

for ns in openchoreo-control-plane openchoreo-observability-plane openchoreo-data-plane; do
  printf '%-34s %s\n' "$ns:" \
    "$(kubectl get svc gateway-default -n $ns \
        -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"
done

Record	Type	Points to
`console`, `api-amp`, `thunder`, `cp`, `api` `.${BASE_DOMAIN}` (or `*.${BASE_DOMAIN}`)	A / CNAME	control-plane gateway address
`traces.${BASE_DOMAIN}`	A / CNAME	observability-plane gateway address
`*.agents.${BASE_DOMAIN}` and `agents.${BASE_DOMAIN}`	A / CNAME	data-plane gateway address

Use A records for IPs (GKE, AKS) and CNAME records for hostnames (EKS). Verify resolution before continuing:

dig +short thunder.${BASE_DOMAIN} traces.${BASE_DOMAIN} test.${AGENTS_DOMAIN}

Phase 2: Agent Manager Installation

With OpenChoreo and Thunder running, you can now install the Agent Manager components — the API, console, and extensions that provide the AI agent management capabilities.

Production — managed PostgreSQL

The Agent Manager chart below deploys an in-cluster PostgreSQL with a default password by default. For production, use a managed database (RDS, Cloud SQL, Azure Database) and add these values to the wso2-agent-manager install in Step 2:

  --set postgresql.enabled=false \
  --set postgresql.external.host="your-db.example.com" \
  --set postgresql.external.port=5432 \
  --set postgresql.external.database=agentmanager \
  --set postgresql.external.username=agentmanager \
  --set postgresql.external.existingSecret=amp-db-credentials \
  --set postgresql.external.existingSecretPasswordKey=password

Create the amp-db-credentials Secret in ${AMP_NS} beforehand. Sizing, backups, and HA then come from your database service.

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

Agent Manager Core : Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
Extensions : Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.

Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables THUNDER_PUBLIC_URL, THUNDER_INTERNAL_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URL, CONSOLE_PUBLIC_HOST, API_PUBLIC_HOST, OBS_API_PUBLIC_HOST, and INSTRUMENTATION_URL must be set from the Configuration Variables section. The *_PUBLIC_HOST variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

Core Components

Install these in order — each depends on the one before it.

Step 1: Gateway Operator

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

helm install gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.6.0 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartVersion=1.1.1 \
  --timeout 600s

Wait for the operator to be ready:

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF

info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

Step 2: Agent Manager (API + Console + PostgreSQL)

The core platform: a Go API server, a React web console, and a PostgreSQL database.

helm install amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --timeout 1800s

Wait for all components:

# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s

Verify

kubectl get pods -n ${AMP_NS}
# Expected: amp-postgresql-0 (Running), amp-api-xxx (Running), amp-console-xxx (Running)

Step 3: Agent Sandbox Module

Agents run as sandboxed pods managed by the Agent Sandbox controller (SandboxTemplate / SandboxWarmPool resources) instead of plain Deployments. This module is required — without it, agent deployments cannot be rendered.

helm upgrade --install agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6

Wait for the controller:

kubectl wait -n agent-sandbox-system \
  --for=condition=available \
  --timeout=180s \
  deployment/agent-sandbox-controller

Verify

kubectl get crd \
  sandboxtemplates.extensions.agents.x-k8s.io \
  sandboxwarmpools.extensions.agents.x-k8s.io \
  sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources

Stronger isolation tiers

With this module, agents run sandboxed under the standard runc runtime. Optionally, individual environments can run agents under gVisor or Kata Containers for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the gVisor and Kata Containers isolation tier guides.

Step 4: Platform Resources

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the container registry endpoint used by build workflows to push agent images.

helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --timeout 1800s

Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at host.k3d.internal:10082. For other environments, override the registry settings:

# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s

Value	Default	Description
`global.registry.endpoint`	`host.k3d.internal:10082`	Registry endpoint for pushing images
`global.baseDomain`	`""`	When set, registry endpoint becomes `registry.<baseDomain>`
`global.defaultResources.registry.tlsVerify`	`false`	Enable TLS verification for registry connections

Extensions

These can be installed in any order after Core is ready.

Step 5: Observability Extension (Agent Manager Observer)

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

helm install amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s

Step 6: Evaluation Extension

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s

info

The default publisher.apiKey must match publisherApiKey.value in the Agent Manager chart. Both default to amp-internal-api-key.

Step 7: API Platform Gateway Extension

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. Install this last — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

helm install api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s

After the gateway is running, apply the OTEL trace collection RestApi:

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml

Verify

kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"

Wire the Remaining Public Endpoints

Two settings are not covered by the shared installation steps above — apply them once all Phase 2 charts are installed.

External AI gateway endpoint — register the cp. hostname on the control-plane gateway route and let the console render the right setup commands:

helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --reuse-values \
  --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \
  --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}"

Deployed-agent endpoints — point the default Environment's gateway at the public agents domain (the Environment's setting wholly replaces the data plane's, so without this override agent invoke URLs are built against a placeholder host) and route the workload publisher's token calls in-cluster:

helm upgrade amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set environment.gateway.http.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.http.port=443 \
  --set environment.gateway.https.host="${AGENTS_DOMAIN}" \
  --set environment.gateway.https.port=443

Verify and Access the Platform

Run a full status check to confirm everything is running:

# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'

Via the Plane Gateways

Everything is hostname-routed through the three plane gateway LoadBalancers, over TLS:

Service	URL
Agent Manager Console	`https://${CONSOLE_PUBLIC_HOST}`
Agent Manager API	`https://${API_PUBLIC_HOST}`
Thunder (OAuth login)	`https://${THUNDER_PUBLIC_HOST}`
Agent Manager Observer	`https://${OBS_API_PUBLIC_HOST}`
OpenChoreo API	`https://api.${BASE_DOMAIN}`
Gateway control plane	`https://${CP_GW_PUBLIC_HOST}`

Open the Console, log in, then create → build → deploy an agent; its invoke URL is published under https://<org>-<project>.${AGENTS_DOMAIN}.

Port-forwarding remains available as a debugging fallback for any individual service (all Agent Manager services are ClusterIP), e.g. kubectl port-forward -n wso2-amp svc/amp-console 3000:3000.

Default credentials: admin / admin

Appendix: Installing Without a Domain (nip.io)

For evaluation on a cluster where you control no DNS zone, nip.io turns LoadBalancer IPs into resolvable hostnames. The flow is the same as above with four differences.

1. Self-signed issuer. In Step 3, create a self-signed chain under the same openchoreo-ca name instead of an ACME or corporate issuer:

kubectl apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: openchoreo-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: openchoreo-ca
  secretName: openchoreo-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: openchoreo-ca
spec:
  ca:
    secretName: openchoreo-ca-secret
EOF

Browsers will warn on every hostname (import openchoreo-ca-secret's ca.crt into your trust store to avoid this), and the Control Plane install in Step 5 needs one extra override because it cannot verify Thunder's JWKS against an untrusted chain: add --set-string openchoreoApi.config.security.authentication.jwt.jwks.skip_tls_verify=true (and --set-string security.oidc.jwksUrlTlsInsecureSkipVerify=true on the Observability Plane install in Step 8).

2. Hostnames come from the first LoadBalancer. The control-plane gateway must exist before you can derive a base domain, but Thunder needs the final hostnames at install time. So: run Step 5's namespace creation, then install only the kgateway-provisioned gateway by running the Control Plane install once with placeholder hostnames (api.placeholder.tld, issuer https://thunder.placeholder.tld, gateway.tls.enabled=false), wait for the LoadBalancer, and derive the base domain:

CP_LB_IP=$(kubectl get svc gateway-default -n openchoreo-control-plane \
  -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export BASE_DOMAIN="${CP_LB_IP//./-}.nip.io"

Now export the Plan Your Deployment variables from this BASE_DOMAIN, continue with Thunder (Step 4), and rerun the Step 5 install command — with the real values — as helm upgrade.

3. One caveat on the shared base domain. All hostnames (console., traces., *.agents. …) resolve to the control-plane gateway's IP, but traces. and *.agents. must reach the other two gateways. Point them correctly by using per-gateway nip.io domains instead: derive TRACES/AGENTS hostnames from the observability and data-plane LoadBalancer IPs after those planes install, e.g. export OBS_API_PUBLIC_HOST="traces.$(kubectl get svc gateway-default -n openchoreo-observability-plane -o jsonpath='{.status.loadBalancer.ingress[0].ip}' | tr . -).nip.io" and the equivalent AGENTS_DOMAIN from the data-plane gateway — then use those values in the observability/data-plane certificates and in Phase 2.

4. Skip Step 10 (DNS records) — nip.io resolution is automatic.

The Thunder reinstall warning applies with full force here: if you tear down and recreate the cluster, the LoadBalancer IP — and therefore every hostname — changes, and Thunder must be reinstalled with the new values.

Cloud Provider Notes

AWS EKS

LoadBalancers return a hostname instead of an IP — use dig to resolve

For internet-facing access, annotate LoadBalancer services:

service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

Ensure security groups allow HTTP/HTTPS traffic

Google Cloud Platform (GKE)

LoadBalancers return IPs directly — no special handling needed
Ensure firewall rules allow HTTP/HTTPS traffic to LoadBalancers

Microsoft Azure (AKS)

LoadBalancers return IPs directly — no special handling needed
Ensure Network Security Groups allow HTTP/HTTPS traffic

Rancher Desktop / k3s

Remove Traefik before installation (see Step 1)
Single-node clusters work for development but may run low on resources with all observability modules
LoadBalancer IPs are assigned via the built-in k3s servicelb
cgroup pids controller issue — see Build workflow fails with cgroup pids error in Troubleshooting

Cleanup

Remove all Agent Manager and OpenChoreo resources:

# 1. Delete plane registrations
kubectl delete clusterdataplane default -n default
kubectl delete clusterworkflowplane default -n default
kubectl delete observabilityplane default -n default

# 2. Uninstall all Helm releases
helm uninstall amp -n wso2-amp
helm uninstall api-platform-default-default -n openchoreo-data-plane
helm uninstall amp-thunder-extension -n amp-thunder
helm uninstall amp-observability-traces -n openchoreo-observability-plane
helm uninstall amp-evaluation-extension -n openchoreo-workflow-plane
helm uninstall amp-platform-resources -n default
helm uninstall gateway-operator -n openchoreo-data-plane
helm uninstall openchoreo-observability-plane -n openchoreo-observability-plane
helm uninstall openchoreo-workflow-plane -n openchoreo-workflow-plane
helm uninstall openchoreo-data-plane -n openchoreo-data-plane
helm uninstall openchoreo-control-plane -n openchoreo-control-plane
helm uninstall openbao -n openbao
helm uninstall external-secrets -n external-secrets
helm uninstall cert-manager -n cert-manager

# 3. Delete namespaces
kubectl delete namespace wso2-amp amp-thunder \
  openchoreo-observability-plane openchoreo-workflow-plane \
  openchoreo-data-plane openchoreo-control-plane \
  openbao external-secrets cert-manager

Production Considerations

The main flow above already covers real domains, trusted wildcard TLS, and gateway-served endpoints. Beyond it, plan for:

Managed dependencies — apply the Production callouts inline above: external PostgreSQL, OpenBao without dev mode, OpenSearch persistence
Identity provider — the seeded admin/admin user is for bootstrap only; connect your organization's IdP or manage users in Thunder before exposing the platform
High availability — deploy multiple replicas across availability zones and adjust resource requests/limits to workload
Security hardening — network policies, scoped RBAC, pod security standards
Certificate operations — with Let's Encrypt, renewals are automatic; monitor cert-manager and keep the DNS-provider credentials Secret rotated

Troubleshooting

LoadBalancer not getting external IP

kubectl describe svc <service-name> -n <namespace>

For EKS, ensure the AWS Load Balancer Controller is installed and the service has the correct annotations.

On k3s/Rancher Desktop, check if another service (like Traefik) is already using the required ports:

kubectl get svc -A --field-selector spec.type=LoadBalancer

Certificate not being issued

kubectl describe certificate <cert-name> -n <namespace>
kubectl get clusterissuers
kubectl get certificaterequests -n <namespace>

Plane registration issues

kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane

Agent Manager API returns 401 for environment/gateway calls

This typically means the OpenChoreo Control Plane's OIDC issuer does not match the iss claim in Thunder-issued JWTs. Verify:

# Check what issuer Thunder puts in tokens
kubectl exec -n amp-thunder deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'

# Check what the Control Plane expects
kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
  | grep issuer

Both must match exactly. If they don't, update the Control Plane's security.oidc.issuer to match Thunder's issuer.

Console shows "refused to connect" on login

The console redirects the browser to Thunder for OAuth login at THUNDER_PUBLIC_URL. Check that the thunder.${BASE_DOMAIN} DNS record points at the control-plane gateway LoadBalancer and that the Thunder HTTPRoute exists:

dig +short thunder.${BASE_DOMAIN}
kubectl get httproute -n openchoreo-control-plane amp-thunder-extension
curl -s -o /dev/null -w '%{http_code}\n' https://thunder.${BASE_DOMAIN}/.well-known/openid-configuration

If you need to change Thunder's public URL after installation, you must uninstall, delete the PVC, and reinstall:

helm uninstall amp-thunder-extension -n amp-thunder
kubectl delete pvc -n amp-thunder --all
# Then reinstall with the new THUNDER_PUBLIC_URL

OpenSearch connectivity issues

kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200

Build pods fail with MOUNT_ATTR_IDMAP / idmap mounts errors

If every agent build fails at pod-sandbox creation with events like:

failed to create containerd container: snapshotter "overlayfs" doesn't support
idmap mounts on this host, configure `slow_chown` to allow a slower and expensive fallback

OCI runtime create failed: ... failed to set MOUNT_ATTR_IDMAP on
/var/lib/kubelet/pods/.../volumes/kubernetes.io~projected/...: invalid argument
(maybe the filesystem used doesn't support idmap mounts on this kernel?)

the build pods are requesting a Linux user namespace (hostUsers: false — the build's root is unprivileged on the node), which requires idmapped-mount support for every filesystem the pod mounts. tmpfs (used by projected service-account token volumes) only gained that support in Linux 6.3 — see the Kubernetes user namespaces documentation. Common node images below that line: Ubuntu 22.04 (kernel 5.15) and Amazon Linux 2 (kernel 5.10). Older Kubernetes versions silently ignored hostUsers, so the same manifests may have "worked" on an older cluster without ever using the feature.

Two fixes:

Preferred: use build nodes with kernel ≥ 6.3 (Ubuntu 24.04, Amazon Linux 2023, recent COS releases).
Otherwise: run builds without user namespaces by setting the Platform Resources chart value:

helm upgrade amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --reuse-values \
  --set buildWorkflows.userNamespaces=false

Note that containerd's slow_chown option alone is not sufficient — it only covers the overlayfs snapshotter, and the build then fails on the tmpfs/projected-volume mount instead.

Agent deploys fail with "Secret does not exist" after an OpenBao restart

This guide installs OpenBao in dev mode (in-memory storage). Any restart of the openbao-0 pod — including node drains and reboots — wipes every secret it holds. Already-running agents keep working (their secrets were injected at startup), but any new deployment or environment promotion fails with ExternalSecrets stuck in SecretSyncedError / Secret does not exist.

Fix: re-save the affected agent's environment variables in the console (agent configuration → environment variables → re-enter and save). The values are rewritten to OpenBao and the ExternalSecret syncs within its refresh interval. For production, configure OpenBao with persistent storage.

Build workflow fails with cgroup pids error (Rancher Desktop)

If the build workflow fails with:

Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available
Error: exit status 126

This happens on Rancher Desktop because the underlying Lima VM (Alpine Linux) does not delegate the pids cgroup controller to containers. The Podman containers inside the build workflow cannot create the required cgroup namespace.

Fix: Patch the ClusterWorkflowTemplates that use Podman to inject a containers.conf that disables cgroup management.

note

The patch commands below require python3 to be installed on your machine.

Run these commands to patch each template:

# Patch gcp-buildpacks-build (build-image step)
kubectl get clusterworkflowtemplate gcp-buildpacks-build -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# Patch publish-image
kubectl get clusterworkflowtemplate publish-image -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''set -e

# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = script.replace('set -e\n', fix, 1)
json.dump(data, sys.stdout)
" | kubectl apply -f -

# Patch amp-generate-workload
kubectl get clusterworkflowtemplate amp-generate-workload -o json | \
  python3 -c "
import json, sys
data = json.load(sys.stdin)
script = data['spec']['templates'][0]['container']['args'][0]
fix = '''# Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround)
cat > /tmp/containers.conf <<CCONF
[engine]
cgroup_manager = \"cgroupfs\"
events_logger = \"file\"
[containers]
pids_limit = 0
CCONF
export CONTAINERS_CONF=/tmp/containers.conf

'''
data['spec']['templates'][0]['container']['args'][0] = fix + script
json.dump(data, sys.stdout)
" | kubectl apply -f -

After patching, re-trigger the build workflow. These patches are applied in-cluster and will be overwritten if the Helm chart (amp-platform-resources) is reinstalled.

note

This issue affects Rancher Desktop specifically because it runs k3s inside a Lima VM with Alpine Linux, which uses OpenRC instead of systemd. The pids cgroup controller is not delegated to containers by default. Other Kubernetes distributions (EKS, GKE, AKS) are not affected.

What You Will Get​

Prerequisites​

Cluster Requirements​

Supported Providers​

Required Tools​

Permissions​

Plan Your Deployment​

1. Hostnames​

2. TLS certificates​

3. Dependencies​

Phase 1: OpenChoreo Platform​

Step 1: Install Cluster Prerequisites​

Step 2: Setup Secrets Store (OpenBao)​

Step 3: Setup TLS Issuer​

Step 4: Install Thunder Extension (Identity Provider)​

Step 5: Install OpenChoreo Control Plane​

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)​

Step 6: Setup Data Plane​

Step 7: Setup Workflow Plane​

Step 8: Setup Observability Plane​

Step 9: Verify OpenChoreo Installation​

Step 10: Publish DNS Records​

Phase 2: Agent Manager Installation​

Core Components​

Step 1: Gateway Operator​

Step 2: Agent Manager (API + Console + PostgreSQL)​

Step 3: Agent Sandbox Module​

Step 4: Platform Resources​

Container registry configuration

Extensions​

Step 5: Observability Extension (Agent Manager Observer)​

Step 6: Evaluation Extension​

Step 7: API Platform Gateway Extension​

Wire the Remaining Public Endpoints​

Verify and Access the Platform​

Via the Plane Gateways​

Appendix: Installing Without a Domain (nip.io)​

Cloud Provider Notes​

Cleanup​

Production Considerations​

Troubleshooting​

What You Will Get

Prerequisites

Cluster Requirements

Supported Providers

Required Tools

Permissions

Plan Your Deployment

1. Hostnames

2. TLS certificates

3. Dependencies

Phase 1: OpenChoreo Platform

Step 1: Install Cluster Prerequisites

Step 2: Setup Secrets Store (OpenBao)

Step 3: Setup TLS Issuer

Step 4: Install Thunder Extension (Identity Provider)

Step 5: Install OpenChoreo Control Plane

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)

Step 6: Setup Data Plane

Step 7: Setup Workflow Plane

Step 8: Setup Observability Plane

Step 9: Verify OpenChoreo Installation

Step 10: Publish DNS Records

Phase 2: Agent Manager Installation

Core Components

Step 1: Gateway Operator

Step 2: Agent Manager (API + Console + PostgreSQL)

Step 3: Agent Sandbox Module

Step 4: Platform Resources

Extensions

Step 5: Observability Extension (Agent Manager Observer)

Step 6: Evaluation Extension

Step 7: API Platform Gateway Extension

Wire the Remaining Public Endpoints

Verify and Access the Platform

Via the Plane Gateways

Appendix: Installing Without a Domain (nip.io)

Cloud Provider Notes

Cleanup

Production Considerations

Troubleshooting