# On Your Environment Set up a production-grade, single-cluster Agent Manager deployment on an existing Kubernetes cluster — AWS EKS, Google GKE, Azure AKS, or any distribution with LoadBalancer support. Each step carries its production configuration inline, so following the guide top to bottom produces a deployment you can put in front of real users. The [Production Reference](#production-reference) at the end is the checklist to audit against, plus the operational concerns that belong to no single step. Just want to evaluate? The [Quick Start Guide](/agent-manager/docs/next/get-started/quick-start/.md) installs everything in a single command using a dev container with k3d, the [k3d guide](/agent-manager/docs/next/guides/on-k3d/.md) covers a local cluster install, and the [VM guide](/agent-manager/docs/next/guides/on-a-vm/.md) covers a single-machine install. Use this page when you are deploying to a real cluster. Stronger agent isolation tiers Agents run sandboxed under the standard **runc** runtime by default. Agent Manager also supports stronger per-environment isolation tiers — **gVisor** (userspace kernel) and **Kata Containers** (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) setup guides. ## What You Will Get[​](#what-you-will-get "Direct link to What You Will Get") Agent Manager is a two-layer system installed in two phases: * **Phase 1 — OpenChoreo (base layer):** [OpenChoreo](https://openchoreo.dev) is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a **Control Plane** for API and configuration, a **Data Plane** for running workloads and gateways, a **Workflow Plane** for builds and CI pipelines, and an **Observability Plane** for traces, logs, and metrics via OpenSearch. * **Phase 2 — Agent Manager :** The AI agent management platform installed on top of OpenChoreo. It includes the **Console** (web UI), **AMP API** (backend), **AI Gateway**, **PostgreSQL** (database), **Secrets Extension** (OpenBao for runtime secret injection), **Agent Manager Observer** (traces, logs, and metrics), and **Evaluation Engine** (automated agent evaluations). This guide installs both layers on your existing Kubernetes cluster, production-first: a real domain, trusted TLS certificates issued automatically by cert-manager, and every management surface served through the three OpenChoreo plane gateways — the only LoadBalancers the platform needs. The OpenChoreo API itself is not exposed outside the cluster; Agent Manager reaches it over cluster DNS. Where a step has a development-grade default and a production one — the databases, secret storage, observability retention — it offers both as tabs. Pick **Production** throughout unless you are deliberately evaluating, and pay attention to [the decisions you cannot change later](#3-decisions-you-cannot-change-later). No domain yet? The main flow assumes you control a DNS zone. If you are evaluating on a cluster without one, follow the [nip.io appendix](#appendix-installing-without-a-domain-nipio) instead — same steps, with hostnames derived from LoadBalancer IPs and self-signed certificates. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") ### Cluster Requirements[​](#cluster-requirements "Direct link to Cluster Requirements") | Requirement | Minimum | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kubernetes version | 1.32+ | | Nodes | 3 | | CPU per node | 4 cores | | RAM per node | 8 GB | | LoadBalancer support | Required | | Default StorageClass | Required, and it must actually provision volumes. On EKS check both halves: clusters created at 1.30+ ship **no default class at all**, and the stock `gp2` class (default on upgraded clusters) references the in-tree provisioner removed in Kubernetes 1.27, so without the EBS CSI driver its PVCs are accepted and stay `Pending` forever. Install the EBS CSI driver and mark a CSI-backed class (e.g. `gp3`) as default | | Node kernel (build nodes) | Linux **6.3+** for user-namespaced agent builds (idmapped-mount support). On older kernels (Ubuntu 22.04's 5.15, Amazon Linux 2's 5.10) set `buildWorkflows.userNamespaces=false` on the Platform Resources chart — see [Troubleshooting](#build-pods-fail-with-mount_attr_idmap--idmap-mounts-errors) | | NetworkPolicy-enforcing CNI | Recommended. The Observability and Evaluation extension charts each ship a `NetworkPolicy` (collector ingress, evaluation-job egress) — inert on a CNI that doesn't implement the NetworkPolicy API. EKS's default VPC CNI, AKS, and legacy GKE don't enforce it out of the box; enable your provider's policy add-on (or install Calico/Cilium) if you want these to actually take effect. k3s (local/CI) enforces it by default. | ### Supported Providers[​](#supported-providers "Direct link to Supported Providers") * **Amazon Web Services** (EKS) * **Google Cloud Platform** (GKE) * **Microsoft Azure** (AKS) * Any Kubernetes distribution with LoadBalancer support ### Required Tools[​](#required-tools "Direct link to Required Tools") | Tool | Version | Purpose | | -------------------------------------------------- | ------------------- | ---------------------------------------- | | [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+ | Kubernetes CLI | | [Helm](https://helm.sh/docs/intro/install/) | v3.12+, **v3 only** | Kubernetes package manager | | curl / dig | — | DNS resolution of LoadBalancer hostnames | ``` kubectl version --client && helm version ``` Helm 4 is not supported by this guide Helm 4 applies charts with Kubernetes server-side apply, where every field has an owner. Several charts below (OpenBao, cert-manager) have controllers that rewrite their own webhook `caBundle` at runtime under a different field manager, so any **re-run** of an install command — including the retries this guide's own troubleshooting recommends — fails with `Apply failed with 1 conflict`. Use Helm 3, or append `--server-side=false` to every `helm install`/`helm upgrade` in this guide. Verify LoadBalancer provisioning works before starting — the alternative is discovering it an hour into the install, when the first plane gateway hangs ``: ``` kubectl create deploy lbtest --image=nginx && kubectl expose deploy lbtest --port=80 --type=LoadBalancer kubectl wait --for=jsonpath='{.status.loadBalancer.ingress}' svc/lbtest --timeout=5m && echo "LoadBalancer OK" kubectl delete svc/lbtest deploy/lbtest ``` ### Permissions[​](#permissions "Direct link to Permissions") You need sufficient privileges to: * Create namespaces, deploy Helm charts * Create LoadBalancer services (only the OpenChoreo plane gateways need them — the Agent Manager services are ClusterIP and are served through those gateways) * Manage cert-manager Issuers and Certificates * Create CRDs and ClusterRoles *** ## Plan Your Deployment[​](#plan-your-deployment "Direct link to Plan Your Deployment") Three decisions shape the installation. Make them now — everything below is driven by the variables you export here. ### 1. Hostnames[​](#1-hostnames "Direct link to 1. Hostnames") Pick a base domain you control (a delegated subdomain like `amp.yourdomain.com` works well). Every management surface is a hostname on one of the three plane gateway LoadBalancers — after the platform installs, you publish DNS records against just those three targets: | Hostname | Serves | DNS record → | | ----------------- | -------------------------------------------------------------------- | ------------------------------ | | `console.` | Agent Manager Console | control-plane gateway LB | | `api-amp.` | Agent Manager API (browser, `amctl`, MCP) | control-plane gateway LB | | `thunder.` | Thunder OAuth login (+ per-environment `-.thunder.`) | control-plane gateway LB | | `cp.` | Gateway control plane (external AI gateways, optional) | control-plane gateway LB | | `traces.` | Agent Manager Observer | observability-plane gateway LB | | `*.agents.` | Deployed-agent invocation endpoints (wildcard) | data-plane gateway LB | Concretely: `console`/`api-amp`/`thunder`/`cp` all point at the control-plane gateway (four records, or a single `*.` wildcard), `traces` points at the observability-plane gateway, and `*.agents.` plus the `agents.` apex point at the data-plane gateway. [Step 10](#step-10-publish-dns-records) lists the exact records once the LoadBalancer addresses exist. ``` export VERSION="0.0.0-dev" export HELM_CHART_REGISTRY="ghcr.io/wso2" export AMP_NS="wso2-amp" export BUILD_CI_NS="openchoreo-workflow-plane" export OBSERVABILITY_NS="openchoreo-observability-plane" export DEFAULT_NS="default" export DATA_PLANE_NS="openchoreo-data-plane" export THUNDER_NS="amp-thunder" # Your base domain — everything below derives from it export BASE_DOMAIN="amp.yourdomain.com" export CONSOLE_PUBLIC_HOST="console.${BASE_DOMAIN}" export API_PUBLIC_HOST="api-amp.${BASE_DOMAIN}" export THUNDER_PUBLIC_HOST="thunder.${BASE_DOMAIN}" export CP_GW_PUBLIC_HOST="cp.${BASE_DOMAIN}" export OBS_API_PUBLIC_HOST="traces.${BASE_DOMAIN}" export AGENTS_DOMAIN="agents.${BASE_DOMAIN}" export CONSOLE_PUBLIC_URL="https://${CONSOLE_PUBLIC_HOST}" export API_PUBLIC_URL="https://${API_PUBLIC_HOST}" export THUNDER_PUBLIC_URL="https://${THUNDER_PUBLIC_HOST}" export OBS_API_PUBLIC_URL="https://${OBS_API_PUBLIC_HOST}" # In-cluster URL backend services use for Thunder JWKS/token calls export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090" # Trace-export endpoint shown to agent developers (see the OTLP note below) export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel" ``` Hostname shape The certificates below are single-level wildcards (`*.`), so every management hostname must sit directly under the base domain — no second-level names like `api.amp.`. ### 2. TLS certificates[​](#2-tls-certificates "Direct link to 2. TLS certificates") The main flow uses **cert-manager with Let's Encrypt (DNS-01)**: publicly trusted certificates, automatic renewal, and wildcard support — you only need API credentials for the DNS provider hosting your zone. Step 3 also shows a corporate-CA alternative. Certificate issuance needs only the DNS zone, so it works before the DNS records for the gateways exist. ### 3. Decisions you cannot change later[​](#3-decisions-you-cannot-change-later "Direct link to 3. Decisions you cannot change later") Each install step below carries its production configuration inline, so you can follow the guide top to bottom and end up with a production deployment. Most of those settings can also be applied later against a running platform. **These cannot** — they are written to a database or a certificate on first boot, and changing them afterwards means an uninstall and reinstall of that component: | Decision | Fixed at | Why it is frozen | | --------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Thunder's database (in-pod SQLite vs external PostgreSQL) | [Step 4](#step-4-install-thunder-extension-identity-provider) | Thunder's OAuth clients are seeded into its database by a `pre-install` hook. Pointing Thunder at a different database later gives you an empty one — no clients, no logins — and `helm upgrade` does not re-seed it | | Thunder's platform client secrets | [Step 4](#step-4-install-thunder-extension-identity-provider) | Same seeding job; changing a secret afterwards means re-running the bootstrap and updating every consumer in one window | | `THUNDER_PUBLIC_URL` and `CONSOLE_PUBLIC_URL` | [Step 4](#step-4-install-thunder-extension-identity-provider) | The issuer URL and the console's OAuth redirect URIs are persisted on first boot | | The MCP resource identifiers (`API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`) | [Step 4](#step-4-install-thunder-extension-identity-provider) | Registered as OAuth resource servers by the same seeding job; an identifier is matched exactly, so a later change makes MCP logins fail with `invalid_target` | | Thunder replica count > 1 | [Step 4](#step-4-install-thunder-extension-identity-provider) | Requires external PostgreSQL **and** a shared Redis cache — the default in-pod SQLite cannot be shared across pods | | Agent Manager's database | [Phase 2](#phase-2-agent-manager-installation) | Switching later is a data migration, not a configuration change | | Each environment's gateway topology | [Phase 2](#phase-2-agent-manager-installation) | A gateway's ingress/egress role is written at first registration and never rewritten, so a single-gateway environment cannot be split later — the environment has to be deleted and recreated | Everything else — OpenSearch sizing, the container registry, gateway hardening, resource quotas, replica counts, isolation tiers — can be changed on a running platform. The [Production Reference](#production-reference) at the end summarises all of it. ### 4. Platform secrets[​](#4-platform-secrets "Direct link to 4. Platform secrets") The charts ship well-known placeholder secrets for the platform's internal OAuth clients and the OpenSearch admin user. Generate real ones now — several are consumed by more than one chart, and the steps below reference these variables by name: ``` export AMP_API_CLIENT_SECRET="$(openssl rand -hex 32)" export AMP_SYSTEM_CLIENT_SECRET="$(openssl rand -hex 32)" export AMP_PUBLISHER_CLIENT_SECRET="$(openssl rand -hex 32)" export AM_OBSERVER_CLIENT_SECRET="$(openssl rand -hex 32)" export WORKFLOW_PUBLISHER_SECRET="$(openssl rand -hex 32)" export OBSERVER_READER_SECRET="$(openssl rand -hex 32)" export OPENSEARCH_USERNAME="admin" export OPENSEARCH_PASSWORD="$(openssl rand -base64 24)" ``` | Variable | Set on | Also consumed by | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | `AMP_API_CLIENT_SECRET` | Thunder ([Step 4](#step-4-install-thunder-extension-identity-provider)) | Agent Manager, API Platform Gateway extension (Phase 2) | | `AMP_SYSTEM_CLIENT_SECRET` | Thunder ([Step 4](#step-4-install-thunder-extension-identity-provider)) | Agent Manager (Phase 2), OpenBao seed ([Step 2](#step-2-set-up-the-secret-stores)) | | `AMP_PUBLISHER_CLIENT_SECRET` | Thunder ([Step 4](#step-4-install-thunder-extension-identity-provider)) | OpenBao seed — read by evaluation workflows at runtime | | `AM_OBSERVER_CLIENT_SECRET` | Thunder ([Step 4](#step-4-install-thunder-extension-identity-provider)) | Observability extension (Phase 2) | | `WORKFLOW_PUBLISHER_SECRET`, `OBSERVER_READER_SECRET` | Thunder ([Step 4](#step-4-install-thunder-extension-identity-provider)) **and** OpenBao seed ([Step 2](#step-2-set-up-the-secret-stores)) | OpenChoreo workflow plane and observer | | `OPENSEARCH_USERNAME` / `OPENSEARCH_PASSWORD` | OpenBao seed ([Step 2](#step-2-set-up-the-secret-stores)) | Observability Plane and logs module ([Step 8](#step-8-setup-observability-plane)) | Keep this shell session for the whole install, and store the values in your secret manager — Thunder's are frozen after [Step 4](#step-4-install-thunder-extension-identity-provider). OTLP trace ingest `INSTRUMENTATION_URL` is the trace-export endpoint shown to agent developers; the traffic rides the data-plane gateway's `/otel` route. Agents running inside the cluster can use the default route hostname `http://default-default.gateway.localhost:19080/otel` (resolvable in-cluster). For agents running outside the cluster, install the API Platform Gateway extension with a public hostname whose DNS points at the data-plane gateway LoadBalancer — `--set gateway.vhost=https://otel.${BASE_DOMAIN}` and `--set gateway.hostname=otel.${BASE_DOMAIN}` — and set `INSTRUMENTATION_URL=https://otel.${BASE_DOMAIN}/otel`. External AI gateways AI gateways running outside this cluster connect through the control-plane gateway at `https://${CP_GW_PUBLIC_HOST}`. Phase 2 passes this hostname to the Agent Manager install; nothing else is needed beyond the DNS record. See [Register an AI Gateway](/agent-manager/docs/next/guides/register-ai-gateway/.md). *** ## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform") OpenChoreo organises its infrastructure into four planes, each handling a different concern: * **Control Plane** — API server and configuration management for the platform * **Data Plane** — runs deployed workloads and API gateways * **Workflow Plane** — builds and CI pipelines for agent deployments * **Observability Plane** — trace, log, and metrics collection via OpenSearch This phase also installs Thunder (the identity provider) as a prerequisite, since the Control Plane and Observability Plane require Thunder's OIDC endpoints for JWT validation. **Estimated time: \~20-30 minutes** (varies by cluster and network). ### Step 1: Install Cluster Prerequisites[​](#step-1-install-cluster-prerequisites "Direct link to Step 1: Install Cluster Prerequisites") **Gateway API CRDs (v1.4.1)** — standard Kubernetes resources for managing network gateways and routing: ``` kubectl apply --server-side --force-conflicts \ -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml ``` info The `--force-conflicts` flag is needed if your cluster already has Gateway API CRDs managed by another controller (e.g., Traefik on k3s/Rancher Desktop). **cert-manager (v1.19.2)** — automates TLS certificate issuance and renewal: ``` helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.19.2 \ --set crds.enabled=true \ --set startupapicheck.timeout=5m \ --wait --timeout 360s ``` **External Secrets Operator (v1.3.2)** — syncs secrets from external stores (like OpenBao) into Kubernetes: ``` helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \ --namespace external-secrets \ --create-namespace \ --version 1.3.2 \ --set installCRDs=true \ --wait --timeout 180s ``` **kgateway (v2.2.1)** — the network gateway for OpenChoreo planes: ``` helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \ --create-namespace \ --namespace openchoreo-control-plane \ --version v2.2.1 helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \ --namespace openchoreo-control-plane \ --create-namespace \ --version v2.2.1 \ --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true ``` note Evaluating on Rancher Desktop / k3s? See the [Rancher Desktop / k3s appendix](#appendix-rancher-desktop--k3s) — Traefik must be removed before this step. ### Step 2: Set Up the Secret Stores[​](#step-2-set-up-the-secret-stores "Direct link to Step 2: Set Up the Secret Stores") Every secret the platform stores is reached through the External Secrets Operator, so the backend behind it is a deployment choice rather than something the platform hardcodes. Each OpenChoreo plane names its own store in `secretStoreRef`, which is what lets one plane use a different backend from another. **OpenBao is required, and this guide installs it.** Agent Manager reads Git credentials for private repositories from OpenBao directly over the Vault API, not through External Secrets, so a store that does not speak that API cannot serve them. That read is what backs the repository and branch pickers in the console. **The data-plane store is yours to choose.** Deployed agents' environment variables are pushed and pulled entirely through External Secrets, so the data plane can name whichever store you prefer. Both of these are supported: * **One store for everything.** The data plane also uses `default`, so OpenBao holds every secret on the platform. This is what the rest of this guide assumes, and it is a perfectly good production answer — nothing below is required. * **A separate store for the data plane.** Point it at any [provider External Secrets supports](https://external-secrets.io/latest/provider/aws-secrets-manager/) — a cloud provider's secret manager, a separate Vault or OpenBao instance, or anything else in that list. See [using a different secret store for deployed agents](#using-a-different-secret-store-for-deployed-agents) below. Worth being precise about one thing: OpenBao is itself an External Secrets provider, reached through ESO's `vault` provider like any other backend. The Git-credential read described above is the *only* path on the platform that bypasses External Secrets — so "OpenBao" and "an External Secrets provider" are not alternatives, and choosing a different data-plane store changes the backend, not the mechanism. | Secrets | Stored for | Backend | | ------------------------------------------------------ | -------------------------------------------------- | ----------------------------------------- | | Platform OAuth client secrets, OpenSearch credentials | Control, workflow and observability planes | OpenBao (seeded below) | | Git credentials for private repositories | Workflow plane, and read directly by Agent Manager | **OpenBao — required** | | Build-time secrets (registry push, workload publisher) | Workflow plane | OpenBao | | Deployed agents' environment variables | Data plane | OpenBao, or any External Secrets provider | * Production * Evaluation only Install with the chart defaults — persistent file storage on a PVC, sealed at rest: ``` helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \ --namespace openbao \ --create-namespace \ --version 0.25.6 \ --set server.dataStorage.size=10Gi \ --timeout 180s # A sealed OpenBao never reports Ready, and there is no pod condition that # means "running but sealed" — conditions are set before the container starts, # so waiting on one races the exec below ('container not found ("openbao")'). # Poll bao itself: exit 2 = sealed but responding (the state we want here); # exit 0 = already unsealed (a re-run); anything else = not up yet. until kubectl exec -n openbao openbao-0 -- bao status >/dev/null 2>&1; do [ $? -eq 2 ] && break sleep 5 done ``` Initialize and unseal. **Store the unseal keys and root token in your secret manager immediately** — they are displayed once, and without them the data on that PVC is unrecoverable: ``` kubectl exec -n openbao openbao-0 -- bao operator init -key-shares=5 -key-threshold=3 # Record the 5 unseal keys and the initial root token, then unseal with any 3: kubectl exec -n openbao openbao-0 -- bao operator unseal kubectl exec -n openbao openbao-0 -- bao operator unseal kubectl exec -n openbao openbao-0 -- bao operator unseal ``` A sealed OpenBao must be unsealed again after every pod restart. For unattended restarts, configure [auto-unseal](https://openbao.org/docs/concepts/seal/) against your cloud KMS through the `server.standalone.config` seal stanza. Create the KV v2 mount, the Kubernetes auth role the platform expects, and the seeded secrets (dev mode does this automatically; a sealed installation does not): ``` kubectl exec -n openbao openbao-0 -- sh -c " export BAO_ADDR=http://127.0.0.1:8200 export BAO_TOKEN='' bao secrets enable -path=secret -version=2 kv bao auth enable kubernetes bao write auth/kubernetes/config kubernetes_host=\"https://\$KUBERNETES_PORT_443_TCP_ADDR:443\" bao policy write openchoreo-secret-reader-policy - < @1.1.1.1 # must return your DNS provider's nameservers ``` ``` kubectl create secret generic route53-credentials \ --namespace cert-manager \ --from-literal=secret-access-key='' kubectl apply -f - < secretAccessKeySecretRef: name: route53-credentials key: secret-access-key selector: dnsZones: - "${BASE_DOMAIN}" EOF ``` tip Test with the Let's Encrypt staging server (`https://acme-staging-v02.api.letsencrypt.org/directory`) first if you expect to iterate — the production server rate-limits duplicate certificates to 5 per week. Prefer ambient credentials over a static access key On EKS, cert-manager picks up IAM Roles for Service Accounts or Pod Identity credentials automatically — no long-lived access key in a cluster Secret. Bind a role scoped to `route53:ChangeResourceRecordSets`/`ListResourceRecordSets` on your one hosted zone (plus `route53:GetChange` and `ListHostedZonesByName`) to the cert-manager ServiceAccount, and reduce the solver to just `region` and `hostedZoneID`. See [cert-manager's Route 53 documentation](https://cert-manager.io/docs/configuration/acme/dns01/route53/) for the equivalent on other setups. If your organization runs an internal PKI, store its intermediate CA certificate and key as a Secret and create a CA issuer. Clients inside your network trust the certificates via your existing root-CA distribution. ``` kubectl create secret tls corporate-ca-secret \ --namespace cert-manager \ --cert=/path/to/intermediate-ca.crt \ --key=/path/to/intermediate-ca.key kubectl apply -f - <<'EOF' apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: openchoreo-ca spec: ca: secretName: corporate-ca-secret EOF ``` For evaluation without a domain, the [nip.io appendix](#appendix-installing-without-a-domain-nipio) creates a self-signed chain under the same `openchoreo-ca` name. Browsers will warn on every hostname and backend components need TLS-verification overrides — do not use this beyond evaluation. ### Step 4: Install Thunder Extension (Identity Provider)[​](#step-4-install-thunder-extension-identity-provider "Direct link to Step 4: Install Thunder Extension (Identity Provider)") Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder. The browser reaches it at `https://${THUNDER_PUBLIC_HOST}` through the control-plane gateway. This step's settings are permanent Thunder writes its issuer URL, the console client's redirect URIs, and every platform OAuth client into its database on **first boot**, seeded by a `pre-install` hook that `helm upgrade` never re-runs. The database, the secrets, and the hostnames below are all effectively frozen once this command completes — getting any of them wrong means uninstalling, discarding Thunder's data, and reinstalling. Confirm your values against [Plan Your Deployment](#plan-your-deployment) before running it. **Database.** Production installations use an external PostgreSQL. Thunder keeps three logical databases — `configdb`, `runtimedb`, and `userdb` — which can live on one server; create them and a role that owns all three before installing, then **load Thunder's schema into each one**. Nothing in the chart creates the tables for PostgreSQL: the init container only initialises the bundled SQLite files, so an empty database makes the pre-install hook fail. On a managed service (RDS, Cloud SQL, Azure Database) the admin user is **not a superuser**, and PostgreSQL 16+ only lets it create a database *owned by another role* if it holds membership in that role. Grant it first, or `CREATE DATABASE ... OWNER thunder` fails with `must be able to SET ROLE "thunder"`: ``` CREATE ROLE thunder LOGIN PASSWORD '...'; GRANT thunder TO CURRENT_USER; -- required on managed PostgreSQL; harmless elsewhere CREATE DATABASE configdb OWNER thunder; -- likewise runtimedb, userdb ``` The same applies to the Agent Manager database role in [Phase 2](#phase-2-agent-manager-installation). * External PostgreSQL * In-pod SQLite (evaluation only) Load the schema first. The scripts ship inside the Thunder image, so copy them out of the version the chart pins and apply each to its database as the owning role: ``` THUNDER_IMAGE="ghcr.io/thunder-id/thunderid:0.45.0" docker create --name thunder-schema "${THUNDER_IMAGE}" for db in configdb runtimedb userdb; do docker cp "thunder-schema:/opt/thunderid/dbscripts/${db}/postgres.sql" "./thunder-${db}.sql" psql "postgresql://thunder@your-db.example.com:5432/${db}" -v ON_ERROR_STOP=1 -f "./thunder-${db}.sql" done docker rm thunder-schema ``` Expect 17 tables in `configdb`, 8 in `runtimedb`, and 5 in `userdb`. If the databases are empty when you install, the `amp-thunder-extension-setup` hook fails with `Server failed to start within 60 seconds` and its pod is deleted, so `kubectl logs` shows nothing — the real error, visible only by running the binary by hand, is `relation "INBOUND_CLIENT" does not exist`. Then store the database password in a Secret and reference it, so it never enters Helm release history: ``` kubectl create namespace ${THUNDER_NS} --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic thunder-db-credentials \ --namespace ${THUNDER_NS} \ --from-literal=password='' export THUNDER_DB_HOST="your-db.example.com" cat > thunder-db-values.yaml < thunder-db-values.yaml ``` Install Thunder with the hostnames, the database, and the platform client secrets generated in [Platform secrets](#4-platform-secrets): ``` helm install amp-thunder-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \ --version ${VERSION} \ --namespace ${THUNDER_NS} \ --create-namespace \ --set thunder.ocIngress.hostname="${THUNDER_PUBLIC_HOST}" \ --set thunder.configuration.server.publicUrl="${THUNDER_PUBLIC_URL}" \ --set thunder.configuration.jwt.issuer="${THUNDER_PUBLIC_URL}" \ --set thunder.configuration.gateClient.hostname="${THUNDER_PUBLIC_HOST}" \ --set thunder.configuration.gateClient.scheme=https \ --set thunder.configuration.gateClient.port=443 \ --set "thunder.configuration.cors.allowedOrigins={${CONSOLE_PUBLIC_URL}}" \ --set "thunder.bootstrap.ampConsoleClient.redirectUris={${CONSOLE_PUBLIC_URL}/login}" \ --set thunder.bootstrap.agentManagerMcpBaseUrl="${API_PUBLIC_URL}" \ --set thunder.bootstrap.observerMcpBaseUrl="${OBS_API_PUBLIC_URL}" \ --set thunder.bootstrap.ampApiClient.clientSecret="${AMP_API_CLIENT_SECRET}" \ --set thunder.bootstrap.ampSystemClient.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \ --set thunder.bootstrap.ampPublisherClient.clientSecret="${AMP_PUBLISHER_CLIENT_SECRET}" \ --set thunder.bootstrap.amObserverClient.clientSecret="${AM_OBSERVER_CLIENT_SECRET}" \ --set thunder.bootstrap.workloadPublisherClient.clientSecret="${WORKFLOW_PUBLISHER_SECRET}" \ --set thunder.bootstrap.observerResourceReaderClient.clientSecret="${OBSERVER_READER_SECRET}" \ --values thunder-db-values.yaml \ --timeout 1800s kubectl wait --for=condition=Available \ deployment -l app.kubernetes.io/instance=amp-thunder-extension \ -n ${THUNDER_NS} --timeout=300s ``` Override all six client secrets The chart bootstraps **six** OAuth clients, and every one it is not given keeps its shipped default. `workloadPublisherClient` and `observerResourceReaderClient` are easy to miss because their secrets are also seeded into OpenBao in [Step 2](#step-2-set-up-the-secret-stores) — but seeding OpenBao only tells the *consumer*; Thunder still registers the default unless the two `--set` lines above are present. Leaving them out breaks the platform in two ways at once. The defaults stay valid, so an install that looks hardened still accepts `openchoreo-workload-publisher-secret` and `openchoreo-observer-resource-reader-client-secret`. And because the consumers read the *generated* values from OpenBao, they present a secret Thunder does not have: every agent build fails at the workload-publish step with `Failed to get access token`, from a `401 invalid_client` that never reaches the build log. When verifying, note the two client families use different token-endpoint auth methods — the four `amp*` clients use HTTP Basic, these two use POST-body credentials. Testing with the wrong method returns `400`, not `401`, which is easy to misread as a rejected credential. MCP resource identifiers `agentManagerMcpBaseUrl` and `observerMcpBaseUrl` become the OAuth resource-server identifiers Thunder registers for the `/mcp` endpoints. Point them at the externally reachable origins of the API and observer — the same values as `agentManagerService.config.serverPublicURL` and `amObserver.publicUrl` in Phase 2 — because Thunder rejects an authorize request with `invalid_target` unless the `resource` an MCP client asks for matches a registered identifier exactly. See the [MCP server reference](/agent-manager/docs/next/reference/mcp-server/.md) for the full set of settings that must move together. Scaling Thunder `thunder.deployment.replicaCount` can only exceed 1 when Thunder runs on external PostgreSQL **and** a shared cache — the default in-memory cache would leave each replica with its own session state. Configure Redis via `thunder.configuration.cache` per the [Thunder caching guide](https://thunderid.dev/docs/v1.0.x/deployment/production-guidelines/#configure-caching), then raise the replica count on this same install. Changing these values later A `helm upgrade` will not change the issuer in issued tokens, the registered redirect URIs, the MCP resource identifiers, or any client secret — those live in Thunder's database, seeded by the pre-install bootstrap job. To change them you must uninstall the chart and discard its data: **delete the PVC** if you used SQLite, or **drop and recreate the three databases** if you used PostgreSQL, then reinstall. Anything already issued against the old issuer stops validating. Discarding Thunder's data also **re-creates the organization**, and Agent Manager keys its own tenant data on that organization's identifier. Every organization-scoped row it holds — agents, deployments, API keys, gateways, monitors, the per-environment Thunder registration — is left pointing at the previous identifier and becomes invisible. The platform stays up and gives no error: the console reports deployment failures, agent identity resolution fails with `env-thunder not provisioned for this environment`, and the gateway cannot re-register because the environment still holds its ingress slot while the organization's gateway list reads empty. Treat a Thunder reinstall as a platform-data reset, not a component restart. On anything but a fresh install, discard Agent Manager's database in the same operation and re-run [env-Thunder provisioning](#provision-thunder-identity-provider-for-the-default-environment) and the gateway extension afterwards. Verify ``` kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \ wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \ | grep -o '"issuer":"[^"]*"' # Expected: "issuer":"${THUNDER_PUBLIC_URL}" (must match your THUNDER_PUBLIC_URL value) ``` ### Step 5: Install OpenChoreo Control Plane[​](#step-5-install-openchoreo-control-plane "Direct link to Step 5: Install OpenChoreo Control Plane") Because the hostnames are fixed up front, the Control Plane installs once with its final configuration. Create the gateway's wildcard certificate first (issuance needs only the DNS zone, not the DNS records): ``` kubectl create namespace openchoreo-control-plane --dry-run=client -o yaml | kubectl apply -f - kubectl apply -f - <-.thunder.${BASE_DOMAIN}`. Without `*.thunder.${BASE_DOMAIN}` the gateway serves a certificate that cannot validate for those names: routing succeeds, but every TLS-verifying client fails. Install the Control Plane with hostnames, TLS, and Thunder OIDC. The OpenChoreo API is deliberately **not** published on the gateway (`openchoreoApi.http.enabled=false`): Agent Manager and the observer call it over cluster DNS, so exposing it would only widen the platform's attack surface. ``` helm upgrade --install openchoreo-control-plane \ oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \ --version 1.1.1 \ --namespace openchoreo-control-plane \ --create-namespace \ --values - < -o jsonpath='{.spec.ports[*].port}{"\n"}' # Expected: 80 443 ``` Register the Data Plane: ``` CA_CERT=$(kubectl get secret cluster-agent-tls \ -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d) kubectl apply -f - </dev/null); do kubectl rollout status "${sts}" -n openchoreo-observability-plane --timeout=900s done ``` With Thunder ≥ 0.45, apply the same entitlement-claim patch from Step 5 to the observer (without it, build-log queries return `403 Access denied`): ``` patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g") echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f - kubectl rollout restart deployment/observer -n openchoreo-observability-plane kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s ``` Install observability modules (logs, metrics, tracing). The logs module is the chart that deploys OpenSearch itself, so its values decide how much history the platform can hold: ``` # Logs module — deploys OpenSearch helm upgrade --install observability-logs-opensearch \ oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \ --create-namespace \ --namespace openchoreo-observability-plane \ --version 0.4.1 \ --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \ --set adapter.openSearchSecretName="opensearch-admin-credentials" \ --set openSearch.persistence.size=100Gi \ --set-string "openSearch.extraEnvs[0].name=OPENSEARCH_INITIAL_ADMIN_PASSWORD" \ --set-string "openSearch.extraEnvs[0].value=${OPENSEARCH_PASSWORD}" \ --timeout 10m # Enable Fluent Bit log collection # On GKE, disable the managed logging agent first — see the note below helm upgrade observability-logs-opensearch \ oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \ --namespace openchoreo-observability-plane \ --version 0.4.1 \ --reuse-values \ --set fluent-bit.enabled=true \ --timeout 10m # Metrics module helm upgrade --install observability-metrics-prometheus \ oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \ --create-namespace \ --namespace openchoreo-observability-plane \ --version 0.6.1 \ --timeout 10m # Tracing module (uses the custom OTel Collector ConfigMap) helm upgrade --install observability-traces-opensearch \ oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \ --create-namespace \ --namespace openchoreo-observability-plane \ --version 0.4.1 \ --set openSearch.enabled=false \ --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \ --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \ --timeout 10m ``` GKE: free port 2020 before enabling Fluent Bit Fluent Bit runs with `hostNetwork: true` (its Kubernetes filter reads from the node-local kubelet) and binds its monitoring server to **port 2020** on every node. GKE's built-in `fluentbit-gke` logging agent already occupies that port, so all Fluent Bit pods stay in `CrashLoopBackOff` with `Cannot listen on 0.0.0.0:2020` — and the only visible symptom is that logs never reach OpenSearch. The platform ships its own log pipeline, so disable GKE's redundant one before the upgrade above: ``` gcloud container clusters update --zone --logging=NONE ``` Turning off `hostNetwork` instead is not a workaround — the filter then cannot reach the kubelet and logs fail with `kubelet upstream connection error`. Sizing and the admin password Two values above differ from the chart defaults. `openSearch.persistence.size` raises the volume from its 8Gi default — size it to your retention needs, since this single-node instance holds every trace, log, and metric the platform stores. The `OPENSEARCH_INITIAL_ADMIN_PASSWORD` override makes OpenSearch's own admin password match the one seeded into OpenBao in [Step 2](#step-2-set-up-the-secret-stores); without it OpenSearch keeps the chart's published default while every client authenticates with yours, and all observability queries fail. Note the capital **S** in `openSearch` — it is the subchart alias, and lowercase `opensearch.*` values are silently ignored. Both are storage-layer choices you can revisit later: growing the volume needs a storage class with expansion enabled, and moving to a managed OpenSearch is a reindex. See [Production Reference](#observability-storage) for the external-OpenSearch route. Register the Observability Plane and link it to other planes: ``` OP_CA_CERT=$(kubectl get secret cluster-agent-tls \ -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d) kubectl apply -f - < The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order: 1. **Agent Manager Core :** Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it. 2. **Extensions :** Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension. Prerequisites Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables `THUNDER_PUBLIC_URL`, `THUNDER_INTERNAL_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`, `CONSOLE_PUBLIC_HOST`, `API_PUBLIC_HOST`, `OBS_API_PUBLIC_HOST`, and `INSTRUMENTATION_URL` must be set from the Configuration Variables section. The production steps below additionally use `AGENTS_DOMAIN` — the domain deployed agents are served on, `agents.` — to build the gateway's hostname and virtual host. The `*_PUBLIC_HOST` variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways. *** ### Core Components[​](#core-components "Direct link to Core Components") Install these in order — each depends on the one before it. #### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator") Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane. The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator: ``` kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f - openssl rand 32 > gateway-aesgcm.key kubectl create secret generic gateway-encryption-keys \ --namespace ${DATA_PLANE_NS} \ --from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key rm -f gateway-aesgcm.key ``` The key is required, and the file name is fixed The controller looks for a specific path, `/app/data/aesgcm-keys/default-aesgcm256-v1.bin`, so the Secret key must be exactly `default-aesgcm256-v1.bin`. Without it the controller crash-loops on `failed to initialize key manager: encryption key file not found for version aesgcm256-v1`, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked. This applies whatever `developmentMode` is set to. Earlier gateway releases auto-generated a key in development mode; from `1.2.0-beta` they do not. Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted. ``` helm install gateway-operator \ oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \ --version 0.10.1 \ --namespace ${DATA_PLANE_NS} \ --set logging.level=info \ --set gatewayApi.installStandardCRDs=false \ --set gateway.helm.chartVersion=1.2.0-beta \ --set gateway.values.gateway.controller.encryptionKeys.enabled=true \ --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \ --timeout 600s ``` Pin the gateway chart, do not inherit it `gateway.helm.chartVersion` decides which gateway chart the operator deploys, and it is the only thing that decides it — the `APIGateway` resource carries no chart version. Operator `0.10.1` defaults to `1.2.0-alpha`, whose templates predate the controller reading its control-plane address from configuration while still shipping `1.2.0-beta` images. The result is a gateway that installs, programs, and serves traffic while never registering with Agent Manager, so it never appears in the gateway list. Pin `1.2.0-beta` as above so the chart and the images match. `gateway.values` is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired. Wait for the operator to be ready: ``` kubectl wait --for=condition=Available \ deployment -l app.kubernetes.io/name=gateway-operator \ -n ${DATA_PLANE_NS} --timeout=300s ``` Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent: ``` kubectl apply -f - < The chart deploys an in-cluster PostgreSQL with a default password unless you point it at an external database. Use a managed service — sizing, backups, failover, and patching then come from your database platform. Create the database and a credentials Secret first: ``` kubectl create namespace ${AMP_NS} --dry-run=client -o yaml | kubectl apply -f - kubectl create secret generic amp-db-credentials \ --namespace ${AMP_NS} \ --from-literal=password='' ``` Then install, wiring the external database and the platform client secrets generated earlier: ``` helm install amp \ oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \ --version ${VERSION} \ --namespace ${AMP_NS} \ --create-namespace \ --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \ --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \ --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \ --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \ --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \ --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \ --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \ --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \ --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \ --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \ --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \ --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \ --set agentManagerService.config.oidc.clientSecret="${AMP_API_CLIENT_SECRET}" \ --set agentManagerService.config.thunder.clientSecret="${AMP_SYSTEM_CLIENT_SECRET}" \ --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \ --set agentManagerService.config.tlsEnabled=true \ --set agentManagerService.replicaCount=2 \ --set agentManagerService.autoscaling.minReplicas=2 \ --set console.replicaCount=2 \ --set agentManagerService.config.openbao.existingSecret=amp-openbao-token \ --set agentManagerService.config.workflowPlaneOpenbao.existingSecret=amp-openbao-token \ --set agentManagerService.config.workflowPlaneOpenbao.existingSecretKey=workflow-plane-openbao-token \ --set postgresql.enabled=false \ --set postgresql.external.host="your-db.example.com" \ --set postgresql.external.port=5432 \ --set postgresql.external.database=agentmanager \ --set postgresql.external.username=agentmanager \ --set postgresql.external.existingSecret=amp-db-credentials \ --set postgresql.external.existingSecretPasswordKey=password \ --set postgresql.external.sslMode=require \ --timeout 1800s ``` Wait for all components — with an external database no PostgreSQL StatefulSet is deployed, so there is nothing to wait for there: ``` # API server kubectl wait --for=condition=Available \ deployment/amp-api -n ${AMP_NS} --timeout=600s # Console kubectl wait --for=condition=Available \ deployment/amp-console -n ${AMP_NS} --timeout=600s ``` `agentManagerService.config.thunder.clientSecret` can instead be supplied through `agentManagerService.config.thunder.existingSecret` (key `thunder-client-secret`) if you prefer to keep it out of Helm release history. `tlsEnabled` decides which agent invoke URL the console publishes `agentManagerService.config.tlsEnabled` defaults to `false`, and it is what selects between an Environment's `http` and `https` endpoint variants when the API builds an agent's invoke URL. Left at the default on an HTTPS platform, the console publishes an `http://` URL — which the browser then refuses to call from an HTTPS page, showing the request as `blocked:mixed-content` with no error anywhere in the platform. The agent itself is healthy and reachable; only the published URL is wrong. Two values that look redundant but are not `agentManagerService.autoscaling.minReplicas=2` is what actually gives the API two replicas. Autoscaling is **on by default** for this component with `minReplicas: 1`, so the HPA takes ownership of the replica count and scales `replicaCount=2` straight back down to one — silently, within seconds. The console has autoscaling off by default, which is why `replicaCount` alone appears to work there. The two `openbao.existingSecret` values point at the Secret minted when you set up the secret stores. Without them the chart falls back to the dev-mode token `root`, which a sealed OpenBao rejects. `workflowPlaneOpenbao` is the one that bites: it is what Agent Manager uses to read Git credentials, so the repository and branch pickers fail and no agent can be created from a private repository, while the platform otherwise looks healthy. Require TLS on the database connection `postgresql.external.sslMode=require` above makes the connection **fail closed**. Left unset, the driver's default of `prefer` applies: it negotiates TLS when the server offers it but silently falls back to an unencrypted connection when it does not, so a misconfigured or replaced instance downgrades without any error. `require` encrypts but does **not** verify the server's identity. To also authenticate the database, use `sslMode=verify-full` with `postgresql.external.sslRootCert`. Set it to `system` only when the certificate chains to a **public** CA; otherwise mount the provider's CA PEM through the chart's `volumes`/`volumeMounts` values and point `sslRootCert` at that path. Managed services differ here — Cloud SQL, for example, issues its server certificate from a Google-managed private CA in either CA mode, so `system` fails with `certificate verify failed` and the CA must be mounted. Check the hostname too: `verify-full` matches the certificate's SAN exactly, so `postgresql.external.host` has to be a name the certificate actually covers rather than an IP. Cloud SQL's SAN carries a trailing dot, which means the host value needs one as well. Enforce it on the server as well, where the client cannot downgrade it: *Require SSL* on Cloud SQL, `rds.force_ssl=1` on RDS, or *require secure transport* on Azure Database for PostgreSQL. Keep the database on a private network so it is never reachable from outside the VPC. info `keyManager.issuer` must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so `agentManagerService.config.oauthAuthorizationServers` only needs setting if it must differ. Likewise the MCP audience entry is derived from `serverPublicURL` above, so `keyManager.audience` only needs setting to change the accepted client IDs. Verify ``` kubectl get pods -n ${AMP_NS} # Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus # amp-postgresql-0 (Running) only when using the in-cluster database ``` #### Step 3: Agent Sandbox Module[​](#step-3-agent-sandbox-module "Direct link to Step 3: Agent Sandbox Module") Agents run as **sandboxed pods** managed by the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io/) controller (`SandboxTemplate` / `SandboxWarmPool` resources) instead of plain Deployments. This module is **required** — without it, agent deployments cannot be rendered. ``` helm upgrade --install agent-sandbox \ oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \ --version 0.1.1 \ --namespace ${DATA_PLANE_NS} \ --create-namespace \ --wait \ --timeout 10m \ --set namespace=openchoreo-control-plane \ --set dataPlaneNamespace=${DATA_PLANE_NS} \ --set dataPlaneServiceAccount=cluster-agent-dataplane \ --set upstream.version=v0.4.6 ``` Wait for the controller: ``` kubectl wait -n agent-sandbox-system \ --for=condition=available \ --timeout=180s \ deployment/agent-sandbox-controller ``` Verify ``` kubectl get crd \ sandboxtemplates.extensions.agents.x-k8s.io \ sandboxwarmpools.extensions.agents.x-k8s.io \ sandboxclaims.extensions.agents.x-k8s.io # All three CRDs should be listed kubectl get clusterrole openchoreo-agent-sandbox-access # RBAC granting the data-plane agent access to sandbox resources ``` Stronger isolation tiers With this module, agents run sandboxed under the standard **runc** runtime. Optionally, individual environments can run agents under **gVisor** or **Kata Containers** for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) isolation tier guides. #### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources") Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the **container registry endpoint** used by build workflows to push agent images. ``` helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \ --set global.oauth.hostHeader="amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local" \ --set global.apiServer.url="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \ --set global.apiServer.hostHeader="openchoreo-api.openchoreo-control-plane.svc.cluster.local" \ --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \ --timeout 1800s ``` These four values default to k3d addresses `global.oauth.tokenUrl` and `global.apiServer.url` default to `http://host.k3d.internal:8080`, which the single-cluster k3d layout reaches through the Docker host and routes by `Host` header. On any other cluster that name does not resolve, and the failure surfaces only when an agent is built: the workflow's `generate-workload` step prints `Failed to get access token:` with an empty body — a connection failure, not an authentication one — and the build ends with `cannot save parameter /mnt/vol/workload-cr.yaml`. Nothing earlier in the install touches these values, so a platform that installs and verifies cleanly still cannot complete a single build until they are set. Point them at the in-cluster services as above. Point the traits at the namespace the gateway is actually in Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from `apiPlatformGateway.namespace`. Leaving it empty derives the per-org-env convention `-`, which is what `add-environment.sh` creates — but [Step 7](#step-7-api-platform-gateway-extension) installs the gateway extension into `${DATA_PLANE_NS}`, matching that chart's own `apiGateway.namespace` default. Set this to the same namespace, as above. Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with `Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known)`, so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a `Transient error` warning in the agent's own log. If you install the gateway extension into `-` instead, leave this unset. ##### Container registry configuration The chart defaults are configured for a local k3d cluster with an in-cluster registry at `host.k3d.internal:10082`. For other environments, override the registry settings: ```bash # Example: external registry with a base domain helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set global.baseDomain="yourdomain.com" \ --set global.defaultResources.registry.tlsVerify=true \ --timeout 1800s # Registry endpoint will be: registry.yourdomain.com # Example: explicit registry endpoint helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set global.registry.endpoint="your-registry.example.com:5000" \ --set global.defaultResources.registry.tlsVerify=true \ --timeout 1800s ``` | Value | Default | Description | | -------------------------------------------- | ------------------------- | ----------------------------------------------------------- | | `global.registry.endpoint` | `host.k3d.internal:10082` | Registry endpoint for pushing images | | `global.baseDomain` | `""` | When set, registry endpoint becomes `registry.` | | `global.defaultResources.registry.tlsVerify` | `false` | Enable TLS verification for registry connections | Not every registry can back build workflows Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly: * **Push-to-create.** Each build pushes its image as `-image` — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. **Amazon ECR does not** and fails every build with `repository ... not found`; the same applies to any registry that requires repositories to be provisioned first. * **Static credentials.** The push step reads a fixed `.dockerconfigjson` from the `registry-push-secret` Secret (or pushes unauthenticated if it is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline. [CNCF Distribution](https://distribution.github.io/distribution/), Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with **no authentication**; put htpasswd in front of it before it carries anything real. *** ### Extensions[​](#extensions "Direct link to Extensions") These can be installed in any order after Core is ready. #### Step 5: Observability Extension (Agent Manager Observer)[​](#step-5-observability-extension-agent-manager-observer "Direct link to Step 5: Observability Extension (Agent Manager Observer)") Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI. `amObserver.observer.idpClientSecret` is the observer's own outbound identity for calls to the OpenChoreo observer, and must match the `am-observer-client` secret seeded into Thunder: ``` helm install amp-observability-traces \ oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \ --version ${VERSION} \ --namespace ${OBSERVABILITY_NS} \ --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \ --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \ --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \ --set amObserver.observer.idpClientSecret="${AM_OBSERVER_CLIENT_SECRET}" \ --set amObserver.replicaCount=2 \ --timeout 1800s kubectl wait --for=condition=Available \ deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s ``` warning `amObserver.auth.issuer` must be the **public** Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and `amctl` send to the Agent Manager API, so its issuer has to match `agentManagerService.config.keyManager.issuer` from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs `JWT validation failed ... invalid issuer`. The chart derives `amObserver.oauth.authorizationServers` and the MCP audience entry from the values above; set them explicitly only if they must differ. #### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension") Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces. ``` helm install amp-evaluation-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \ --version ${VERSION} \ --namespace ${BUILD_CI_NS} \ --timeout 1800s ``` info Evaluation jobs publish scores using the `amp-publisher-client` OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (`secret/amp-publisher-client-secret`) and must match the value set on `thunder.bootstrap.ampPublisherClient.clientSecret`. caution The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the `kubernetes` Service ClusterIP. Check what that address is: ``` kubectl -n default get endpoints kubernetes ``` `networkPolicy.evaluationJob.apiServer.cidrs` defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet: ``` helm install amp-evaluation-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \ --version ${VERSION} \ --namespace ${BUILD_CI_NS} \ --timeout 1800s \ --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=" ``` Setting it is required if `kubernetes` resolves to a **public** endpoint (public GKE/AKS control planes) or to `100.64.0.0/10`. The default will not match those, and every evaluation will publish its scores and then report FAILED. #### Step 7: API Platform Gateway Extension[​](#step-7-api-platform-gateway-extension "Direct link to Step 7: API Platform Gateway Extension") Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange. The bootstrap job authenticates to the Agent Manager API as `amp-api-client`, so it needs the secret you seeded into Thunder. Keep it out of Helm release history with a Secret reference: ``` kubectl create secret generic gateway-idp-credentials \ --namespace ${DATA_PLANE_NS} \ --from-literal=client-id=amp-api-client \ --from-literal=client-secret="${AMP_API_CLIENT_SECRET}" helm install api-platform-default-default \ oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \ --version ${VERSION} \ --namespace ${DATA_PLANE_NS} \ --set agentManager.orgName=default \ --set gateway.environment=default \ --set gateway.type=BOTH \ --set developmentMode=false \ --set gateway.vhost="https://default-default.${AGENTS_DOMAIN}" \ --set gateway.hostname="default-default.${AGENTS_DOMAIN}" \ --set agentManager.idp.existingSecret=gateway-idp-credentials \ --timeout 1800s kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \ -n ${DATA_PLANE_NS} --timeout=300s ``` Set the vhost now — it cannot be changed later `gateway.vhost` and `gateway.hostname` default to the local k3d addresses (`http://default-default.gateway.localhost:19080`). The bootstrap job writes them into Agent Manager at **first registration only**: on every later run it finds the gateway already present, logs `already exists`, and exits without reconciling the record. A `helm upgrade` with corrected values changes nothing, and there is no error to notice — the console simply keeps showing the `.localhost:19080` virtual host, which is what an operator would copy when wiring an external caller. Correcting it after the fact means deleting the gateway registration and re-registering, so get these right on the first install. `gateway.type=BOTH` registers this gateway to serve both inbound and outbound traffic, which is what a single-gateway environment needs. It matches the chart default and is set explicitly here because the role is written once at registration and never rewritten — see [Gateway roles and topology](#gateway-roles-and-topology) for the split alternative and its constraints. `developmentMode=false` turns off the relaxed security checks the chart enables by default. It depends on the encryption key from [Step 1](#step-1-gateway-operator): without one the install fails immediately with `encryptionKeys must be enabled: at-rest encryption is mandatory`. Set it here rather than hardening afterwards, so the gateway is never registered in a relaxed state. The extension already renders an OTLP ingest route for its own gateway (`-otel-restapi` in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace `-`, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace: ``` kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml ``` Verify ``` kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS} # STATUS should show "Programmed" kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap # STATUS should show "Complete" ``` ### Wire the Remaining Public Endpoints[​](#wire-the-remaining-public-endpoints "Direct link to Wire the Remaining Public Endpoints") Two settings are not covered by the shared installation steps above — apply them once all Phase 2 charts are installed. **External AI gateway endpoint** — register the `cp.` hostname on the control-plane gateway route and let the console render the right setup commands: ``` helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \ --version ${VERSION} \ --namespace ${AMP_NS} \ --reuse-values \ --set "agentManagerService.ocIngress.gatewayMgmt.hostnames={${CP_GW_PUBLIC_HOST}}" \ --set console.config.gatewayControlPlaneUrl="https://${CP_GW_PUBLIC_HOST}" ``` **Deployed-agent endpoints** — point the default Environment's gateway at the public agents domain (the Environment's setting wholly replaces the data plane's, so without this override agent invoke URLs are built against a placeholder host) and route the workload publisher's token calls in-cluster: ``` helm upgrade amp-platform-resources oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --reuse-values \ --set global.oauth.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \ --set environment.gateway.http.host="${AGENTS_DOMAIN}" \ --set environment.gateway.http.port=80 \ --set environment.gateway.https.host="${AGENTS_DOMAIN}" \ --set environment.gateway.https.port=443 ``` Both variants, each on its own port The Environment's gateway binding **wholly replaces** the data plane's, so a partial override silently discards what [Step 6](#step-6-setup-data-plane) wrote rather than merging with it. Set both variants, and give each the port its listener actually serves — `80` for `http`, `443` for `https`, matching the plane gateway. Putting `443` on the `http` variant is the trap: the console then publishes `http://:443`, an HTTP scheme on the TLS port, and a browser refuses to call it from the HTTPS console as mixed content. Nothing in the platform reports an error, because the gateway and the agent are both fine. **Environments you add later** — the override above applies to the default Environment only. Environments created afterwards with [`add-environment.sh`](/agent-manager/docs/next/guides/environment-management/.md) read their hostnames from Agent Manager's own config, which still holds the chart's placeholder defaults. Record the same values there so an added environment resolves the way the default one does: ``` helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \ --version ${VERSION} \ --namespace ${AMP_NS} \ --reuse-values \ --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \ --set agentManagerService.config.agentsHttpPort=80 \ --set agentManagerService.config.agentsHttpsPort=443 ``` The two ports must match the `environment.gateway.http.port` and `https.port` you set above — the same one-port-per-listener rule, for the same reason. Skip this and added environments still install, but their agents are published on `am-gateway.localhost`: the console shows an empty invoke URL and try-out returns `405` against its own host. The symptom appears only when you create a second environment, long after this step. Publishing added environments' gateways Each added environment also gets its own API Platform Gateway at `-.`, which carries that environment's OTel ingest and LLM-proxy endpoints. Those stay in-cluster unless you gave the gateway a public hostname (see [`INSTRUMENTATION_URL`](#1-hostnames)) — deployed agents reach the runtime in-cluster either way, so this only matters for callers outside the cluster. If you did, record it too, so added environments publish a reachable URL rather than a `localhost` one. Run this instead of the command above — it carries the same three agent settings plus the gateway ones: ``` helm upgrade amp oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \ --version ${VERSION} \ --namespace ${AMP_NS} \ --reuse-values \ --set agentManagerService.config.agentsBaseDomain="${AGENTS_DOMAIN}" \ --set agentManagerService.config.agentsHttpPort=80 \ --set agentManagerService.config.agentsHttpsPort=443 \ --set agentManagerService.config.gatewayBaseDomain="otel.${BASE_DOMAIN}" \ --set agentManagerService.config.gatewayVhostScheme=https \ --set agentManagerService.config.gatewayVhostPort=443 ``` That needs a `*.otel.${BASE_DOMAIN}` DNS record and gateway certificate coverage — and, per [RFC 4592](https://www.rfc-editor.org/rfc/rfc4592), adding that wildcard makes `otel.${BASE_DOMAIN}` an empty non-terminal that a broader `*.${BASE_DOMAIN}` stops covering, so keep an explicit record for it. This is the same trap the `*.thunder.${BASE_DOMAIN}` record carries. ### Provision Thunder Identity Provider for the Default Environment[​](#provision-thunder-identity-provider-for-the-default-environment "Direct link to Provision Thunder Identity Provider for the Default Environment") Every Environment needs its own dedicated Thunder ID instance. This is separate from the platform Thunder installed in Phase 1, which only handles console and API login. This instance is what issues each agent its own OAuth2 credential (AgentID) in that Environment. Run this once for the default Environment: agents can still be created without it, but they will never get an AgentID there, since there is no Thunder instance to provision one against, and Agent Manager will keep retrying and failing in the background. Run it after Agent Manager (`amp`) is installed and reachable at `API_PUBLIC_URL`, since this step registers the generated credential with the Agent Manager API. ``` curl -fsSL "https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/scripts/add-environment-thunder.sh" \ -o add-environment-thunder.sh ``` Pick the tab matching your [Step 3](#step-3-setup-tls-issuer) issuer choice — the CA trust setting below only differs by that choice, nothing else changes. Pass the platform client secret `IDP_CLIENT_SECRET` defaults to the **shipped** `amp-api-client-secret`, so the script fails with `Could not obtain an access token to call agent-manager-service` on any install that generated real secrets in [Platform secrets](#4-platform-secrets). Passing `${AMP_API_CLIENT_SECRET}` as below is what makes the two steps consistent. The script stores its own generated system-client secret before that call, so a failure here leaves work half-done — rerun it once the credentials are right. ::: (Self-signed evaluation installs: see difference 5 in the [nip.io appendix](#appendix-installing-without-a-domain-nipio).) * Let's Encrypt (DNS-01) * Corporate CA Platform Thunder's certificate comes from a real, publicly-trusted CA, so env-Thunder's container trust store already trusts it — no custom CA bundle needs to be fetched or mounted. ``` ENV_NAME=default \ DISPLAY_NAME="Default" \ ORG_NAME=default \ WAIT_TIMEOUT=300s \ AMP_API_URL="${API_PUBLIC_URL}/api/v1" \ IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \ IDP_CLIENT_ID=amp-api-client \ IDP_CLIENT_SECRET="${AMP_API_CLIENT_SECRET}" \ PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}" \ PLATFORM_THUNDER_JWKS_URL="${THUNDER_PUBLIC_URL}/oauth2/jwks" \ THUNDER_HOST_BASE_DOMAIN="${BASE_DOMAIN}" \ TLS_ENABLED=true \ SKIP_CA_BUNDLE_TRUST=true \ bash add-environment-thunder.sh ``` `SKIP_CA_BUNDLE_TRUST` does **not** apply here — that flag only skips the fetch when omitted, it doesn't mount your own CA. Pass your CA directly instead, extracted live from the running certificate rather than guessed from a secret name (the `corporate-ca-secret` you created in Step 3 stores only `tls.crt`/`tls.key`, not a separate `ca.crt` key): ``` PLATFORM_THUNDER_CA_PEM="$(echo | openssl s_client -connect "${THUNDER_PUBLIC_HOST}:443" \ -servername "${THUNDER_PUBLIC_HOST}" -showcerts 2>/dev/null \ | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{print}')" ``` `PLATFORM_THUNDER_CA_PEM` only gets mounted into env-Thunder's own pod, for its in-cluster JWKS check. It does nothing for the `curl` call this script itself makes from **your** machine to `IDP_TOKEN_URL` a moment later, and that call is just as HTTPS as everything else here. Trust the CA locally too, or that call fails with the same "unable to get local issuer certificate" error a browser would show: ``` CURL_CA_BUNDLE="$(mktemp)" if [ -f /etc/ssl/cert.pem ]; then SYSTEM_CA_BUNDLE=/etc/ssl/cert.pem # macOS elif [ -f /etc/ssl/certs/ca-certificates.crt ]; then SYSTEM_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt # Debian/Ubuntu else echo "Could not find a system CA bundle at /etc/ssl/cert.pem or /etc/ssl/certs/ca-certificates.crt. Find yours and set SYSTEM_CA_BUNDLE to it manually." >&2 exit 1 fi cat "${SYSTEM_CA_BUNDLE}" > "${CURL_CA_BUNDLE}" echo "${PLATFORM_THUNDER_CA_PEM}" >> "${CURL_CA_BUNDLE}" export CURL_CA_BUNDLE ``` Keeping your system bundle in there too (rather than only the one CA) matters because this script also fetches a public CA bundle from `curl.se` over plain HTTPS, and pointing `CURL_CA_BUNDLE` at just your corporate CA would break that unrelated call. ``` ENV_NAME=default \ DISPLAY_NAME="Default" \ ORG_NAME=default \ WAIT_TIMEOUT=300s \ AMP_API_URL="${API_PUBLIC_URL}/api/v1" \ IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \ IDP_CLIENT_ID=amp-api-client \ IDP_CLIENT_SECRET="${AMP_API_CLIENT_SECRET}" \ PLATFORM_THUNDER_ISSUER="${THUNDER_PUBLIC_URL}" \ PLATFORM_THUNDER_JWKS_URL="${THUNDER_PUBLIC_URL}/oauth2/jwks" \ PLATFORM_THUNDER_CA_PEM="${PLATFORM_THUNDER_CA_PEM}" \ THUNDER_HOST_BASE_DOMAIN="${BASE_DOMAIN}" \ TLS_ENABLED=true \ bash add-environment-thunder.sh ``` info Only the Let's Encrypt path can use `SKIP_CA_BUNDLE_TRUST=true` — it's specifically for a certificate that's already backed by a CA every trust store already knows. For a Corporate CA, that flag doesn't help you: setting it to `true` skips the fetch outright, and simply omitting it falls back to the script's own auto-detect, which looks for a secret named `amp-local-root-ca-secret` — a name the Corporate CA tab in Step 3 never creates. `PLATFORM_THUNDER_CA_PEM` sidesteps that entirely by pulling whatever certificate is actually being served, live, regardless of what you named things, but it only solves trust *inside the cluster*. The `CURL_CA_BUNDLE` step above is a separate fix for trust on **your own machine**, where this script's own `curl` calls run. (Running with the self-signed evaluation chain? See the [nip.io appendix](#appendix-installing-without-a-domain-nipio).) Verify ``` kubectl get pods -n amp-thunder-default-default # All pods should be Running kubectl get secret amp-thunder-default-default-admin-credentials \ -n amp-thunder-default-default -o jsonpath='{.data.password}' | base64 -d # Prints the env-Thunder console admin password for this Environment — save it, # it is not shown again ``` Safe to re-run: the system-client secret and admin password are reused, never rotated. ### Point the Gateway at the Environment's Thunder[​](#point-the-gateway-at-the-environments-thunder "Direct link to Point the Gateway at the Environment's Thunder") The gateway extension was installed before this Environment's Thunder existed, so it has no way to validate the tokens that Thunder issues. Wire it now that the issuer is known — this registers Thunder both as a gateway key manager (JWT validation for deployed agents) and as an identity provider visible in the console: ``` export ENV_THUNDER_RELEASE="amp-thunder-default-default" export ENV_THUNDER_ISSUER="https://default-default.thunder.${BASE_DOMAIN}" export ENV_THUNDER_JWKS="http://${ENV_THUNDER_RELEASE}-service.${ENV_THUNDER_RELEASE}.svc.cluster.local:8090/oauth2/jwks" helm upgrade api-platform-default-default \ oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \ --version ${VERSION} \ --namespace ${DATA_PLANE_NS} \ --reuse-values \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].name=agent-manager-service" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].issuer=agent-manager-service" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.uri=http://amp-api.${AMP_NS}.svc.cluster.local:9000/auth/external/jwks.json" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[0].jwks.remote.skipTlsVerify=true" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].name=ThunderKeyManager" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].issuer=${ENV_THUNDER_ISSUER}" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.uri=${ENV_THUNDER_JWKS}" \ --set "apiGateway.config.policyConfigurations.jwtauth_v1.keymanagers[1].jwks.remote.skipTlsVerify=false" \ --set "bootstrap.identityProviders[0].name=ThunderKeyManager" \ --set "bootstrap.identityProviders[0].issuer=${ENV_THUNDER_ISSUER}" \ --set "bootstrap.identityProviders[0].jwksUri=${ENV_THUNDER_JWKS}" \ --set "bootstrap.identityProviders[0].skipTlsVerify=false" \ --timeout 900s ``` Both key managers must be listed. Helm's `--set` on an indexed array replaces the whole list, so re-stating `keymanagers[0]` is what keeps the internal `agent-manager-service` entry that API-key authentication relies on. Without this, agents cannot accept OAuth tokens Skipping this step leaves the gateway with only its internal key manager. Agents still work with API keys, so the platform looks complete — but no agent endpoint can validate a Thunder-issued OAuth token, and the console's gateway page shows *No identity providers configured* with nothing to explain why. The controller restarts, and its volume is single-attach This upgrade rolls the gateway controller. Its data volume is `ReadWriteOnce`, so on a multi-node cluster the replacement pod may be scheduled to a different node and block with `Multi-Attach error for volume`, leaving the `APIGateway` at `Programmed=False` indefinitely. Delete the old controller pod to release the volume; the new one then starts and the gateway returns to `Ready`. ``` kubectl get pods -n ${DATA_PLANE_NS} | grep gateway-controller # If two are listed and one is stuck ContainerCreating, delete the older Running pod kubectl delete pod -n ${DATA_PLANE_NS} ``` *** ## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform") Run a full status check to confirm everything is running: ``` # All pods across key namespaces kubectl get pods -n openchoreo-control-plane kubectl get pods -n openchoreo-data-plane kubectl get pods -n openchoreo-workflow-plane kubectl get pods -n openchoreo-observability-plane kubectl get pods -n wso2-amp kubectl get pods -n amp-thunder # Helm releases helm list -A | grep -E 'openchoreo|amp|gateway' ``` ### Via the Plane Gateways[​](#via-the-plane-gateways "Direct link to Via the Plane Gateways") Everything is hostname-routed through the three plane gateway LoadBalancers, over TLS: | Service | URL | | -------------------------- | -------------------------------- | | **Agent Manager Console** | `https://${CONSOLE_PUBLIC_HOST}` | | **Agent Manager API** | `https://${API_PUBLIC_HOST}` | | **Thunder (OAuth login)** | `https://${THUNDER_PUBLIC_HOST}` | | **Agent Manager Observer** | `https://${OBS_API_PUBLIC_HOST}` | | **Gateway control plane** | `https://${CP_GW_PUBLIC_HOST}` | Open the Console, log in, then create → build → deploy an agent; its invoke URL is published under `https://-.${AGENTS_DOMAIN}`. Port-forwarding remains available as a debugging fallback for any individual service (all Agent Manager services are ClusterIP), e.g. `kubectl port-forward -n wso2-amp svc/amp-console 3000:3000`. **Default credentials:** `admin` / `admin` — bootstrap only; replace before exposing the platform (see [Identity and Access](#identity-and-access)). *** ## Cloud Provider Notes[​](#cloud-provider-notes "Direct link to Cloud Provider Notes") AWS EKS * LoadBalancers return a **hostname** instead of an IP — use `dig` to resolve * For internet-facing access, annotate LoadBalancer services: ``` service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing ``` * Ensure security groups allow HTTP/HTTPS traffic Google Cloud Platform (GKE) * LoadBalancers return IPs directly — no special handling needed * Ensure firewall rules allow HTTP/HTTPS traffic to LoadBalancers Microsoft Azure (AKS) * LoadBalancers return IPs directly — no special handling needed * Ensure Network Security Groups allow HTTP/HTTPS traffic *** ## Production Reference[​](#production-reference "Direct link to Production Reference") Following the steps above with the **Production** tab selected at each choice already gives you a production deployment: real hostnames, trusted wildcard TLS, every surface behind the three plane gateways, the OpenChoreo API kept in-cluster, generated platform secrets, external databases, and sealed secret storage. This section is the other half — the checklist to audit an install against, plus the operational concerns that belong to no single step. Agent Manager runs on [OpenChoreo](https://openchoreo.dev), and for the components OpenChoreo owns its documentation is the authoritative reference; its [production configuration index](https://openchoreo.dev/docs/getting-started/try-it-out/on-your-environment/#production-configuration) collects the relevant pages. This guide targets a **single-cluster** deployment — OpenChoreo's multi-cluster topology material is out of scope here. ### Checklist[​](#checklist "Direct link to Checklist") Work down this table against an existing install. The **When** column matters: the first four rows cannot be corrected without reinstalling the component, so verify them before going live. | Item | Where | When it can be applied | | -------------------------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------- | | Thunder on external PostgreSQL | [Step 4](#step-4-install-thunder-extension-identity-provider) | Install only | | Thunder platform client secrets are not the shipped defaults | [Step 4](#step-4-install-thunder-extension-identity-provider) | Install only | | Thunder issuer and console redirect URIs are the final hostnames | [Step 4](#step-4-install-thunder-extension-identity-provider) | Install only | | Agent Manager on external PostgreSQL | [Phase 2](#phase-2-agent-manager-installation) | Install (migration afterwards) | | OpenBao sealed, persistent, and unseal keys escrowed | [Step 2](#step-2-set-up-the-secret-stores) | Any time — re-seeding required | | OpenSearch volume sized to retention, admin password matches OpenBao | [Step 8](#step-8-setup-observability-plane) | Any time | | Container registry is yours, with `tlsVerify=true` | [Phase 2](#phase-2-agent-manager-installation) | Any time — affects new builds | | Console `admin`/`admin` replaced by real users or a connected IdP | [Identity](#identity-and-access) | Any time | | JWT signature verification active on the API and the observer | [Identity](#identity-and-access) | Any time | | Gateway development mode off with encryption keys configured | [Gateway hardening](#gateway-hardening) | Both are set during installation | | Post-upgrade patches re-applied | [After a chart upgrade](#after-a-chart-upgrade) | After every relevant `helm upgrade` | | Build workloads bounded by a node pool or ResourceQuota | [Build workflows](#build-workflows) | Any time | | Stateless services running multiple replicas | [High availability](#high-availability-and-scaling) | Any time | | NetworkPolicies, RBAC, and Pod Security Standards applied | [Cluster security](#certificates-and-cluster-security) | Any time | | Stronger isolation tier chosen where agents run untrusted code | [Agent isolation](#agent-isolation) | Any time, per environment | ### After a chart upgrade[​](#after-a-chart-upgrade "Direct link to After a chart upgrade") Three settings in this guide are applied with `kubectl` on top of what Helm renders, and **any** `helm upgrade` of the owning chart reverts them — including upgrades that have nothing to do with authentication. The symptoms are quiet: `200` responses with empty lists, or a `403` on build logs, while every pod stays healthy. After upgrading the Control Plane or the Observability Plane, re-apply: | Patch | Owning chart | Symptom when missing | | -------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------ | | `openchoreo-api-config` entitlement claim ([Step 5](#patch-the-service-account-entitlement-claim-required-with-thunder--045)) | Control Plane | Service calls return `200` with empty lists; gateway bootstrap cannot find the environment | | `ClusterAuthzRoleBinding` entitlement claims ([Step 5](#patch-the-service-account-entitlement-claim-required-with-thunder--045)) | Control Plane | Same as above | | `observer-auth-config` entitlement claim ([Step 8](#step-8-setup-observability-plane)) | Observability Plane | Build-log queries return `403` | Two cautions learned the hard way. Because these use `kubectl patch`/`apply`, the field manager takes ownership of the patched fields, so the *next* `helm upgrade` of that chart can fail outright with a conflict such as `conflict with "kubectl-patch" using openchoreo.dev/v1alpha1: .spec.entitlement.claim`. And the obvious recovery — deleting the objects so Helm recreates them — is unsafe for the ClusterAuthzRoleBindings, because the set spans two charts: `amp-platform-resources` owns `amp-api-client-binding` and `amp-observer-reader-binding`, which the Control Plane chart will not recreate. Losing `amp-api-client-binding` silently empties every organization and agent list in the console. Re-create them from their own chart with `helm get manifest amp-platform-resources -n ${DEFAULT_NS}` rather than deleting blindly. ### Identity and Access[​](#identity-and-access "Direct link to Identity and Access") The seeded `admin`/`admin` console login is for bootstrap only. Create real users in Thunder or connect your organization's identity provider before exposing the platform — see ThunderID's [Manage Users](https://thunderid.dev/docs/v1.0.x/guides/users/manage-users/) and [Manage Identity Providers](https://thunderid.dev/docs/v1.0.x/guides/identity-providers/manage-identity-providers/) guides. This is separate from [Configure Identity Providers at the Gateway](/agent-manager/docs/next/guides/configure-identity-providers-at-the-gateway/.md), which covers the issuers a gateway trusts for agent endpoints. Each environment also gets its own Thunder instance, provisioned in Phase 2, which issues agents their OAuth credentials in that environment. Those are separate installations from the platform Thunder; its admin password is generated once at provisioning and shown only in the verify step, so capture it into your secret manager then. Apply the same database and credential hygiene per environment. There is no first-class rotation flow for the platform's internal client secrets yet: because Thunder seeds them through a `pre-install` hook, changing one means re-running the bootstrap and updating every consumer in the same maintenance window. Treat them as install-time values and protect them accordingly. #### Confirm JWT signature verification is active[​](#confirm-jwt-signature-verification-is-active "Direct link to Confirm JWT signature verification is active") Both the Agent Manager API and the observer carry a **development-only** path that decodes a token's payload without verifying its signature. Leaving that active on an exposed deployment would let anyone forge a token, so it is worth confirming explicitly even though it is off by default. Neither service is at risk with the values this guide sets. The API takes the bypass only when its JWKS URL is empty **and** `IS_LOCAL_DEV_ENV` is `true`; that variable defaults to `false`, the chart does not set it on the API deployment, and the install above sets `agentManagerService.config.keyManager.jwksUrl` explicitly. With a JWKS URL configured it always verifies, and with neither configured it rejects the request outright rather than falling back. The observer is guarded in the chart: `amObserver.auth.isLocalDevEnv` defaults to `false`, and while it is false the chart refuses to render unless `auth.jwksUrl`, `auth.issuer`, and `auth.audience` are all set. The checks are therefore that nothing has switched the bypass on, and that the JWKS URLs point at your Thunder: ``` # Neither should report "true" kubectl get deploy amp-api -n ${AMP_NS} \ -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="IS_LOCAL_DEV_ENV")].value}{"\n"}' kubectl get deploy amp-observer -n ${OBSERVABILITY_NS} \ -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="IS_LOCAL_DEV_ENV")].value}{"\n"}' # The JWKS URL should be your Thunder instance kubectl get configmap -n ${AMP_NS} -o yaml | grep -i KEY_MANAGER_JWKS_URL ``` Empty output from the first two commands is the expected, secure result — the variable is simply not set. The `amp-db-migration` job does set `IS_LOCAL_DEV_ENV=true`, which is harmless: it runs the binary with `-migrate -server=false`, so it serves no requests and validates no tokens. ### External Dependencies[​](#external-dependencies "Direct link to External Dependencies") The install steps carry the values; these are the reference pages behind them. | Dependency | Reference | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Thunder identity | [Thunder production guidelines](https://thunderid.dev/docs/v1.0.x/deployment/production-guidelines/) — database, TLS, encryption keys, CORS, [caching](https://thunderid.dev/docs/v1.0.x/deployment/production-guidelines/#configure-caching) | | Secret management | [OpenChoreo secret management](https://openchoreo.dev/docs/platform-engineer-guide/secret-management/), [OpenBao auto-unseal](https://openbao.org/docs/concepts/seal/) | | Observability storage | [OpenChoreo observability and alerting](https://openchoreo.dev/docs/platform-engineer-guide/observability-alerting/) | | Container registry | [OpenChoreo registry configuration](https://openchoreo.dev/docs/platform-engineer-guide/container-registry-configuration/#registry-providers) | | API Platform Gateway | [High-availability production deployment](https://wso2.com/api-platform/docs/api-gateway/1.1.0/deployment/high-availability-production-deployment/), [database configuration](https://wso2.com/api-platform/docs/api-gateway/1.1.0/deployment/production-deployment/database-configuration/) | | AI Gateway | [Kubernetes deployment with the gateway operator](https://wso2.com/api-platform/docs/ai-gateway/1.1.0/deployment-modes/kubernetes/gateway-operator/) | #### Observability storage[​](#observability-storage "Direct link to Observability storage") To move off the in-cluster OpenSearch entirely, point the observability modules at a managed domain instead of letting the logs module deploy one: install it with `openSearch.enabled=false` and set the endpoint and credentials the modules and observer use through the `opensearch-admin-credentials` secret created in [Step 8](#step-8-setup-observability-plane). Existing data does not follow automatically — moving a running platform is a reindex, so prefer deciding this before trace volume accumulates. Metrics come from the in-cluster Prometheus deployed by the metrics module; for retention and alerting beyond its defaults, see the OpenChoreo observability guide above. ### Gateway Hardening[​](#gateway-hardening "Direct link to Gateway Hardening") The gateway extension provisions one gateway per environment and ships with `developmentMode: true`, which relaxes several security checks. The production install in [Phase 2](#phase-2-agent-manager-installation) already sets `developmentMode=false` and provisions the encryption key the gateway needs, so a gateway installed by following this guide starts hardened — there is nothing to turn off afterwards. If you inherited a gateway that was installed with development mode on, turn it off in place: ``` helm upgrade api-platform-default-default \ oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \ --version ${VERSION} \ --namespace ${DATA_PLANE_NS} \ --reuse-values \ --set developmentMode=false \ --timeout 1800s ``` Encryption keys are **not** part of that decision. The gateway requires them whichever way `developmentMode` is set, which is why the key Secret is created before the operator is installed. If the upgrade above fails with `encryptionKeys must be enabled: at-rest encryption is mandatory`, the key is missing — create it and wire it through the operator as [Step 1](#phase-2-agent-manager-installation) describes, then retry. Configure the gateway through the operator, not the rendered ConfigMap Any gateway chart value can be set through the operator's `gateway.values` passthrough, which the operator merges into the values it deploys the gateway with: ``` --set gateway.values.gateway.controller.= ``` The extension also renders a `-config` ConfigMap holding those values, and editing it directly looks like a shortcut. It is not: `kubectl apply` takes ownership of `.data.values.yaml`, so the next `helm upgrade` of the extension fails with a field-manager conflict, and recovery means deleting the ConfigMap so Helm can recreate it. Set values on the operator instead. A gateway that fails its first install stays failed If the gateway stack cannot start on first install — a missing encryption key is the usual cause — the operator exhausts its retry budget and **persists that state**. Correcting the values afterwards changes nothing on its own: the `APIGateway` stays `Programmed=False` with `Max retries (10) exceeded`, through operator restarts and re-reconciles, and its Helm sub-release is left in `pending-install`. Recovery is to clear both halves: `helm uninstall -gw` to remove the wedged sub-release, then delete the `APIGateway` resource and `helm upgrade` the extension to re-render it. It reconciles to `Ready` within a minute once both are gone. Verify the gateway actually registered, rather than only that it is running: ``` kubectl get apigateway -n ${DATA_PLANE_NS} # expect PROGRAMMED=True ``` A gateway can be `Programmed=True` with both pods healthy and still not be registered with Agent Manager — it then never appears in the console's gateway list and cannot serve agent traffic. If that happens, check the controller's startup log for an empty control-plane address, which means the chart and image versions do not match: see the pinning note in [Step 1](#phase-2-agent-manager-installation). Rate-limiting policies default to an **in-memory** backend, which does not share counters across gateway replicas. When running more than one replica, switch to Redis through the extension's `apiGateway.config.policyConfigurations.ratelimit_v1` values (`backend: redis` plus the `redis` connection block). Rolling back the gateway extension needs the API reachable The extension's bootstrap job runs as a `pre-install`, `pre-upgrade` **and `pre-rollback`** hook, so `helm rollback` executes it too and can fail — leaving the release stuck in `pending-rollback` — when the Agent Manager API is unreachable. If you are rolling back precisely because the platform is unhealthy, that is exactly when it will bite. Recovery is to re-run `helm rollback` once the API is back, or `helm rollback --no-hooks` to skip the job entirely. #### Gateway roles and topology[​](#gateway-roles-and-topology "Direct link to Gateway roles and topology") Every gateway declares a role: **`INGRESS`** for traffic arriving at the agents in an environment, **`EGRESS`** for the calls those agents make out to LLMs and MCP servers, or **`BOTH`**. The install above registers the default environment's gateway as `BOTH`, which is what a single-gateway environment needs. Running the two directions on separate gateways lets you scale and harden them independently — an internet-facing ingress gateway and an egress gateway that reaches your model providers are rarely under the same load or the same threat model. That **split topology** is created per environment with `GATEWAY_TOPOLOGY=split`, which installs a second, egress-only release beside the ingress one. See [Gateway topology](/agent-manager/docs/next/guides/environment-management/.md#gateway-topology) for how to create an environment that way, including the shorter name limit it imposes. Two constraints are worth knowing before you choose: * An environment may hold **at most one** ingress-capable gateway (`INGRESS` or `BOTH`). Egress gateways are uncapped. * A gateway's role is fixed when it first registers. A later `helm upgrade` with a different `gateway.type` only logs a drift warning — it never rewrites the role — so an environment's topology cannot be switched in place. Moving an existing environment to a split topology means deleting and recreating it. `REGULAR` and `AI` are still accepted as input, mapping to `BOTH` and `EGRESS`, but they are deprecated; prefer the role names. AI gateways can also run **outside** the cluster (for example, close to a private LLM endpoint), connecting back through the control-plane gateway at `https://${CP_GW_PUBLIC_HOST}`. Register those as `EGRESS`, since they serve outbound traffic only — see [Register an AI Gateway](/agent-manager/docs/next/guides/register-ai-gateway/.md). The WSO2 [AI Gateway Kubernetes deployment guide](https://wso2.com/api-platform/docs/ai-gateway/1.1.0/deployment-modes/kubernetes/gateway-operator/) covers running one under the gateway operator. ### Build Workflows[​](#build-workflows "Direct link to Build Workflows") Agent builds run as Argo workflows in the Workflow Plane, and build pods have **no built-in resource ceiling** — a burst of builds can starve the platform and agent workloads on a shared node. Two mitigations, in order of preference: 1. **Dedicated build nodes** — run the workflow plane's build workloads on a separate node pool so build spikes cannot affect serving workloads. Remember the build-node kernel requirement (Linux 6.3+ for user-namespaced builds, see [Prerequisites](#cluster-requirements)). This is the preferred mitigation: it needs no per-pod resource declarations and cannot reject a build. 2. **Namespace ResourceQuota plus a LimitRange** — cap total build resource consumption. Both objects go in the namespace where builds actually run, which is **`workflows-`** (`workflows-default` for the default environment), not the workflow-plane namespace — that one holds the Argo controller and cluster agent, not the build pods. The `LimitRange` is not optional, and it has to come before the quota. Build pods declare no requests or limits of their own, and a ResourceQuota that tracks `requests.*`/`limits.*` **rejects** every pod that omits them. Applying the quota alone fails each build within seconds at its first step with `must specify limits.cpu for: init,main,wait` — a hard failure, not a queued build. Note all three containers in that message: `main` is the build step, while `init` and `wait` are injected by Argo, so a `LimitRange` is what covers them all from one object you control. **Measure before you set these.** The values below are placeholders, not recommendations — the right numbers depend on what your agents build. Run one build with the `LimitRange` absent and watch what a step actually uses: ``` # While a build is running, in another shell: kubectl top pod -n workflows-default --containers ``` Set `defaultRequest` near the steady-state usage you observe and `default` (the limit) above the peak, with headroom. Then apply both objects: ``` kubectl apply -f - <` is created by OpenChoreo the first time a workflow runs there, so apply these after the first successful build rather than during the install. Repeat them for each environment you add. ### High Availability and Scaling[​](#high-availability-and-scaling "Direct link to High Availability and Scaling") The production install already runs the Agent Manager API, console, and observer at two replicas. Spread them across availability zones with your cluster's default topology spreading, and adjust to load: * `agentManagerService.replicaCount` and `console.replicaCount` on `wso2-agent-manager` * `amObserver.replicaCount` on the observability extension * `thunder.deployment.replicaCount` on the Thunder extension — needs **both** external PostgreSQL (the default in-pod SQLite cannot be shared between pods) and a shared Redis cache. The database half is fixed at install; see [Step 4](#step-4-install-thunder-extension-identity-provider) The Agent Manager API runs a monitor scheduler that coordinates through a lock, so scheduled evaluations fire once regardless of replica count. Stateful dependencies get their availability from the platforms behind them: managed PostgreSQL, managed or clustered OpenSearch, and a sealed OpenBao with auto-unseal configured. Gateway replica guidance is in the gateway HA documentation linked above. ### Certificates and Cluster Security[​](#certificates-and-cluster-security "Direct link to Certificates and Cluster Security") With the Let's Encrypt DNS-01 issuer, certificate renewals are automatic — monitor cert-manager (its `Certificate` resources expose `Ready` and expiry conditions) and rotate the DNS-provider credentials Secret on your normal schedule. With a corporate CA, track the intermediate's expiry yourself. Apply your organization's standard cluster hardening around the platform: restrict namespace-to-namespace traffic with NetworkPolicies — in particular, only the gateways should reach the OTel collector and OpenSearch — scope RBAC for humans and CI to the namespaces they operate in, and enforce Pod Security Standards on the workload namespaces. Confirm your CNI actually **enforces** the `NetworkPolicy` objects the platform ships (the Observability and Evaluation extensions each include one — see [Cluster Requirements](#cluster-requirements)). A cluster without a policy engine selected — EKS's default VPC CNI, AKS, or legacy GKE — accepts those objects silently while blocking nothing, so the protection you think you have is not there. ### Agent Isolation[​](#agent-isolation "Direct link to Agent Isolation") Agents run sandboxed under **runc** by default. Environments that execute untrusted or externally authored agent code can be moved to stronger isolation tiers — [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) (userspace kernel) or [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) (per-agent VM). Both have hardware/OS requirements and need dedicated nodes; plan the node pools alongside the build-node decision above. *** ## Appendix: Installing Without a Domain (nip.io)[​](#appendix-installing-without-a-domain-nipio "Direct link to Appendix: Installing Without a Domain (nip.io)") For evaluation on a cluster where you control no DNS zone, [nip.io](https://nip.io) turns LoadBalancer IPs into resolvable hostnames. This path is **not for production** — browsers warn on every hostname and the certificates are self-signed. Full nip.io / self-signed flow differences The flow is the same as above with five differences. **1. Self-signed issuer.** In Step 3, create a self-signed chain under the same `openchoreo-ca` name instead of an ACME or corporate issuer: ``` kubectl apply -f - <<'EOF' apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-bootstrap spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: openchoreo-ca namespace: cert-manager spec: isCA: true commonName: openchoreo-ca secretName: openchoreo-ca-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-bootstrap kind: ClusterIssuer --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: openchoreo-ca spec: ca: secretName: openchoreo-ca-secret EOF ``` Browsers will warn on every hostname (import `openchoreo-ca-secret`'s `ca.crt` into your trust store to avoid this), and the Control Plane install in Step 5 needs one extra override because it cannot verify Thunder's JWKS against an untrusted chain: add `--set-string openchoreoApi.config.security.authentication.jwt.jwks.skip_tls_verify=true` (and `--set-string security.oidc.jwksUrlTlsInsecureSkipVerify=true` on the Observability Plane install in Step 8). **2. Hostnames come from the first LoadBalancer.** The control-plane gateway must exist before you can derive a base domain, but Thunder needs the final hostnames at install time. So: run Step 5's namespace creation, then install *only* the kgateway-provisioned gateway by running the Control Plane install once with placeholder hostnames (`api.placeholder.tld`, issuer `https://thunder.placeholder.tld`, `gateway.tls.enabled=false`), wait for the LoadBalancer, and derive the base domain: ``` CP_LB_IP=$(kubectl get svc gateway-default -n openchoreo-control-plane \ -o jsonpath='{.status.loadBalancer.ingress[0].ip}') export BASE_DOMAIN="${CP_LB_IP//./-}.nip.io" ``` Now export the [Plan Your Deployment](#plan-your-deployment) variables from this `BASE_DOMAIN`, continue with Thunder (Step 4), and rerun the Step 5 install command — with the real values — as `helm upgrade`. **3. One caveat on the shared base domain.** All hostnames (`console.`, `traces.`, `*.agents.` …) resolve to the control-plane gateway's IP, but `traces.` and `*.agents.` must reach the *other* two gateways. Point them correctly by using per-gateway nip.io domains instead: derive `TRACES`/`AGENTS` hostnames from the observability and data-plane LoadBalancer IPs after those planes install, e.g. `export OBS_API_PUBLIC_HOST="traces.$(kubectl get svc gateway-default -n openchoreo-observability-plane -o jsonpath='{.status.loadBalancer.ingress[0].ip}' | tr . -).nip.io"` and the equivalent `AGENTS_DOMAIN` from the data-plane gateway — then use those values in the observability/data-plane certificates and in Phase 2. **4. Skip Step 10** (DNS records) — nip.io resolution is automatic. **5. Env-Thunder provisioning trusts the self-signed CA explicitly.** In the [Provision Thunder Identity Provider](#provision-thunder-identity-provider-for-the-default-environment) step, `SKIP_CA_BUNDLE_TRUST` doesn't mount anything, and the script's own auto-detect looks for a different, local-dev-specific secret name than the `openchoreo-ca-secret` this appendix creates. Extract the CA live instead: ``` PLATFORM_THUNDER_CA_PEM="$(echo | openssl s_client -connect "${THUNDER_PUBLIC_HOST}:443" \ -servername "${THUNDER_PUBLIC_HOST}" -showcerts 2>/dev/null \ | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{print}')" ``` This only covers env-Thunder's own pod. The `curl` call the script makes from **your** machine to `IDP_TOKEN_URL` needs the same CA trusted locally, or it fails with "unable to get local issuer certificate": ``` CURL_CA_BUNDLE="$(mktemp)" if [ -f /etc/ssl/cert.pem ]; then SYSTEM_CA_BUNDLE=/etc/ssl/cert.pem # macOS elif [ -f /etc/ssl/certs/ca-certificates.crt ]; then SYSTEM_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt # Debian/Ubuntu else echo "Could not find a system CA bundle at /etc/ssl/cert.pem or /etc/ssl/certs/ca-certificates.crt. Find yours and set SYSTEM_CA_BUNDLE to it manually." >&2 exit 1 fi cat "${SYSTEM_CA_BUNDLE}" > "${CURL_CA_BUNDLE}" echo "${PLATFORM_THUNDER_CA_PEM}" >> "${CURL_CA_BUNDLE}" export CURL_CA_BUNDLE ``` Then run the script exactly as in the Corporate CA tab (with `PLATFORM_THUNDER_CA_PEM` set). The Thunder reinstall warning applies with full force here: if you tear down and recreate the cluster, the LoadBalancer IP — and therefore every hostname — changes, and Thunder must be reinstalled with the new values. *** ## Appendix: Rancher Desktop / k3s[​](#appendix-rancher-desktop--k3s "Direct link to Appendix: Rancher Desktop / k3s") For **evaluation** on Rancher Desktop or another k3s distribution — not a production target. k3s-specific setup and workarounds * Single-node clusters work for development but may run low on resources with all observability modules * LoadBalancer IPs are assigned via the built-in k3s servicelb * **cgroup `pids` controller issue** — see [Build workflow fails with cgroup pids error](#build-workflow-fails-with-cgroup-pids-error) in Troubleshooting k3s ships with Traefik which binds to host ports 80/443 and conflicts with OpenChoreo's kgateway. Remove Traefik before [Step 1](#step-1-install-cluster-prerequisites): ``` helm uninstall traefik -n kube-system helm uninstall traefik-crd -n kube-system ``` After removing Traefik, re-apply the Gateway API CRDs (Traefik's CRD chart may have removed them): ``` kubectl apply --server-side --force-conflicts \ -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml ``` *** ## Cleanup[​](#cleanup "Direct link to Cleanup") Remove all Agent Manager and OpenChoreo resources: ``` # 1. Delete plane registrations kubectl delete clusterdataplane default -n default kubectl delete clusterworkflowplane default -n default kubectl delete clusterobservabilityplane default # 2. Uninstall all Helm releases helm uninstall amp -n wso2-amp helm uninstall api-platform-default-default -n openchoreo-data-plane helm uninstall amp-thunder-extension -n amp-thunder helm uninstall amp-observability-traces -n openchoreo-observability-plane helm uninstall amp-evaluation-extension -n openchoreo-workflow-plane helm uninstall amp-platform-resources -n default helm uninstall gateway-operator -n openchoreo-data-plane helm uninstall openchoreo-observability-plane -n openchoreo-observability-plane helm uninstall openchoreo-workflow-plane -n openchoreo-workflow-plane helm uninstall openchoreo-data-plane -n openchoreo-data-plane helm uninstall openchoreo-control-plane -n openchoreo-control-plane helm uninstall openbao -n openbao helm uninstall external-secrets -n external-secrets helm uninstall cert-manager -n cert-manager # 3. Delete namespaces kubectl delete namespace wso2-amp amp-thunder \ openchoreo-observability-plane openchoreo-workflow-plane \ openchoreo-data-plane openchoreo-control-plane \ openbao external-secrets cert-manager ``` *** ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") LoadBalancer not getting external IP ``` kubectl describe svc -n ``` For EKS, ensure the AWS Load Balancer Controller is installed and the service has the correct annotations. On k3s/Rancher Desktop, check if another service (like Traefik) is already using the required ports: ``` kubectl get svc -A --field-selector spec.type=LoadBalancer ``` Certificate not being issued ``` kubectl describe certificate -n kubectl get clusterissuers kubectl get certificaterequests -n ``` Plane registration issues ``` kubectl get clusterdataplane default -n default -o yaml kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane ``` Agent Manager API returns 401 for environment/gateway calls This typically means the OpenChoreo Control Plane's OIDC issuer does not match the `iss` claim in Thunder-issued JWTs. Verify: ``` # Check what issuer Thunder puts in tokens kubectl exec -n amp-thunder deploy/amp-thunder-extension-deployment -- \ wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \ | grep -o '"issuer":"[^"]*"' # Check what the Control Plane expects kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \ | grep issuer ``` Both must match exactly. If they don't, update the Control Plane's `security.oidc.issuer` to match Thunder's issuer. Console shows "refused to connect" on login The console redirects the browser to Thunder for OAuth login at `THUNDER_PUBLIC_URL`. Check that the `thunder.${BASE_DOMAIN}` DNS record points at the control-plane gateway LoadBalancer and that the Thunder HTTPRoute exists: ``` dig +short thunder.${BASE_DOMAIN} kubectl get httproute -n openchoreo-control-plane amp-thunder-extension curl -s -o /dev/null -w '%{http_code}\n' https://thunder.${BASE_DOMAIN}/.well-known/openid-configuration ``` If you need to change Thunder's public URL after installation, you must uninstall, delete the PVC, and reinstall: ``` helm uninstall amp-thunder-extension -n amp-thunder kubectl delete pvc -n amp-thunder --all # Then reinstall with the new THUNDER_PUBLIC_URL ``` Evaluations mysteriously fail or hang with connection errors The evaluation-job NetworkPolicy scopes DNS egress to a specific namespace/pod label (default: `kube-system` / `k8s-app=kube-dns`). If your cluster's DNS runs elsewhere — NodeLocal DNSCache, a relabeled CoreDNS, a managed provider's own DNS add-on — that egress rule won't match it, DNS resolution fails inside the eval-job pod, and *every* other egress rule fails right along with it, since nothing can resolve first. Check this first when evaluations fail with generic connection errors: point `networkPolicy.evaluationJob.dns.namespace`/`podLabel` at your cluster's actual DNS workload. OpenSearch connectivity issues ``` kubectl get pods -n openchoreo-observability-plane -l app=opensearch kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \ curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200 ``` Build pods fail with MOUNT\_ATTR\_IDMAP / idmap mounts errors If every agent build fails at pod-sandbox creation with events like: ``` failed to create containerd container: snapshotter "overlayfs" doesn't support idmap mounts on this host, configure `slow_chown` to allow a slower and expensive fallback ``` or ``` OCI runtime create failed: ... failed to set MOUNT_ATTR_IDMAP on /var/lib/kubelet/pods/.../volumes/kubernetes.io~projected/...: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?) ``` the build pods are requesting a Linux **user namespace** (`hostUsers: false` — the build's root is unprivileged on the node), which requires idmapped-mount support for every filesystem the pod mounts. tmpfs (used by projected service-account token volumes) only gained that support in **Linux 6.3** — see the [Kubernetes user namespaces documentation](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/). Common node images below that line: Ubuntu 22.04 (kernel 5.15) and Amazon Linux 2 (kernel 5.10). Older Kubernetes versions silently ignored `hostUsers`, so the same manifests may have "worked" on an older cluster without ever using the feature. Two fixes: 1. **Preferred:** use build nodes with kernel ≥ 6.3 (Ubuntu 24.04, Amazon Linux 2023, recent COS releases). 2. **Otherwise:** run builds without user namespaces by setting the Platform Resources chart value: ``` helm upgrade amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --reuse-values \ --set buildWorkflows.userNamespaces=false ``` Note that containerd's `slow_chown` option alone is **not** sufficient — it only covers the overlayfs snapshotter, and the build then fails on the tmpfs/projected-volume mount instead. Agent deploys fail with "Secret does not exist" after an OpenBao restart This guide installs OpenBao in **dev mode** (in-memory storage). Any restart of the `openbao-0` pod — including node drains and reboots — wipes every secret it holds. Already-running agents keep working (their secrets were injected at startup), but any **new** deployment or environment promotion fails with ExternalSecrets stuck in `SecretSyncedError` / `Secret does not exist`. Fix: re-save the affected agent's environment variables in the console (agent configuration → environment variables → re-enter and save). The values are rewritten to OpenBao and the ExternalSecret syncs within its refresh interval. For production, configure OpenBao with persistent storage. Build workflow fails with cgroup pids error (Rancher Desktop) If the build workflow fails with: ``` Error: OCI runtime error: crun: the requested cgroup controller `pids` is not available Error: exit status 126 ``` This happens on Rancher Desktop because the underlying Lima VM (Alpine Linux) does not delegate the `pids` cgroup controller to containers. The Podman containers inside the build workflow cannot create the required cgroup namespace. **Fix:** Patch the ClusterWorkflowTemplates that use Podman to inject a `containers.conf` that disables cgroup management. note The patch commands below require **python3** to be installed on your machine. Run these commands to patch each template: ``` # Patch gcp-buildpacks-build (build-image step) kubectl get clusterworkflowtemplate gcp-buildpacks-build -o json | \ python3 -c " import json, sys data = json.load(sys.stdin) script = data['spec']['templates'][0]['container']['args'][0] fix = '''set -e # Fix: disable cgroup management for Podman (Rancher Desktop cgroup pids workaround) cat > /tmp/containers.conf < /tmp/containers.conf < /tmp/containers.conf <