On k3d (Locally)
Install the Agent Manager on a local Kubernetes cluster using k3d — a lightweight tool that runs Kubernetes inside Docker.
The Quick Start Guide installs everything in a single command using a dev container. Use this page when you want to understand the full setup or need to customize it.
Agents run sandboxed under the standard runc runtime by default. Agent Manager also supports stronger per-environment isolation tiers — gVisor (userspace kernel) and Kata Containers (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the gVisor and Kata Containers setup guides.
What You Will Get
Agent Manager is a two-layer system installed in two phases:
-
Phase 1 — OpenChoreo (base layer): OpenChoreo is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a Control Plane for API and configuration, a Data Plane for running workloads and gateways, a Workflow Plane for builds and CI pipelines, and an Observability Plane for traces, logs, and metrics via OpenSearch.
-
Phase 2 — Agent Manager : The AI agent management platform installed on top of OpenChoreo. It includes Thunder (identity provider — installed with OpenChoreo since it provides JWT validation for all planes), the Console (web UI), AMP API (backend), AI Gateway, PostgreSQL (database), Secrets Extension (OpenBao for runtime secret injection), Agent Manager Observer (traces, logs, and metrics), and Evaluation Engine (automated agent evaluations).
This guide installs both layers on a single-node k3d cluster on your local machine.
This setup is for development and exploration. For production deployments, see the Production Considerations section.
Prerequisites
Hardware
| Resource | Minimum |
|---|---|
| RAM | 8 GB |
| CPU | 4 cores |
| Disk | ~10 GB free |
Required Tools
| Tool | Version | Purpose |
|---|---|---|
| Docker | v26.0+ | Container runtime |
| kubectl | v1.32+ | Kubernetes CLI |
| Helm | v3.12+ | Kubernetes package manager |
| k3d | v5.8+ | Local Kubernetes clusters |
Verify all tools:
docker --version && kubectl version --client && helm version && k3d version
If you use Colima instead of Docker Desktop, start a dedicated agent-manager profile so the install does not interfere with your default Colima instance:
colima start --profile agent-manager \
--vm-type=vz --vz-rosetta --network-address \
--cpu 4 --memory 8
--network-address assigns the VM a stable IP so the cluster keeps working after network changes.
Prefix the cluster creation command in Step 1 with K3D_FIX_DNS=0.
Required Ports
The following host ports must be free before installation. The Agent Manager services do not need host ports of their own: the Console and API are served through the Control Plane gateway (8080), the Agent Manager Observer through the Observability Plane gateway (11080), and trace ingestion through the Data Plane gateway (19080).
| Port | Purpose |
|---|---|
| 6550 | Kubernetes API |
| 8080 | Control Plane Gateway (HTTP) — Console, API, Thunder, OpenChoreo API, external AI gateway control plane |
| 8443 | Control Plane Gateway (HTTPS) |
| 19080 | Data Plane Gateway (HTTP) — deployed agents, OTLP /otel ingest |
| 19443 | Data Plane Gateway (HTTPS) |
| 10082 | Container Registry (Workflow Plane) |
| 11080 | Observability Plane Gateway (HTTP) — Agent Manager Observer, Observer API |
| 11082 | OpenSearch API |
| 11085 | OpenSearch HTTPS |
Phase 1: OpenChoreo Platform
OpenChoreo organises its infrastructure into four planes, each handling a different concern:
- Control Plane — API server and configuration management for the platform
- Data Plane — runs deployed workloads and API gateways
- Workflow Plane — builds and CI pipelines for agent deployments
- Observability Plane — trace, log, and metrics collection via OpenSearch
This phase installs all four with Agent Manager-specific configuration. Estimated time: ~15-20 minutes.
Step 1: Create k3d Cluster
Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings:
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/quick-start/k3d-config.yaml \
| k3d cluster create --config=-
Set the kubectl context:
k3d kubeconfig merge amp-local --kubeconfig-merge-default
kubectl config use-context k3d-amp-local
Generate machine IDs for Fluent Bit log collection:
for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do
docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"
done
kubectl cluster-info --context k3d-amp-local
# Should show cluster running at https://0.0.0.0:6550
Step 2: Apply CoreDNS Configuration
Enables *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, and *.gateway.localhost DNS resolution inside the cluster — required for in-cluster lookups of the Console/API/Thunder hostnames as well as agent-to-agent and AI gateway traffic:
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/k8s/coredns-amp-custom.yaml
CoreDNS's reload plugin can miss the override if the ConfigMap mounts after the pod's initial parse, so restart it to make sure the rewrite rules are active before anything depends on them:
kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system --timeout=60s
Step 3: Install Cluster Prerequisites
These are infrastructure components that OpenChoreo depends on. You only install them once per cluster.
Gateway API CRDs — standard Kubernetes resources for managing network gateways and routing:
kubectl apply --server-side \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
cert-manager (v1.19.2) — automates TLS certificate issuance and renewal:
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.19.2 \
--set crds.enabled=true \
--wait --timeout 180s
External Secrets Operator (v1.3.2) — syncs secrets from external stores (like OpenBao) into Kubernetes:
helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
--namespace external-secrets \
--create-namespace \
--version 1.3.2 \
--set installCRDs=true \
--wait --timeout 180s
kgateway (v2.2.1) — the network gateway for OpenChoreo planes:
helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--create-namespace \
--namespace openchoreo-control-plane \
--version v2.2.1
helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
--namespace openchoreo-control-plane \
--create-namespace \
--version v2.2.1 \
--set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
Step 4: Setup Secrets Store (OpenBao)
OpenBao provides secret management for the Workflow Plane and deployed agents:
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
--namespace openbao \
--create-namespace \
--version 0.25.6 \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-openbao.yaml \
--timeout 180s
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
Configure the External Secrets ClusterSecretStore:
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secrets-openbao
namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: default
spec:
provider:
vault:
server: "http://openbao.openbao.svc:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "openchoreo-secret-writer-role"
serviceAccountRef:
name: "external-secrets-openbao"
namespace: "openbao"
EOF
Step 5: Configuration Variables
Set these variables before continuing. They are used by the remaining Phase 1 steps and all of Phase 2.
export VERSION="0.0.0-dev"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"
export OPENCHOREO_INTERNAL_URL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"
export THUNDER_PUBLIC_URL="http://thunder.amp.localhost:8080"
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"
export CONSOLE_PUBLIC_URL="http://console.amp.localhost:8080"
export API_PUBLIC_URL="http://api.amp.localhost:8080"
export OBS_API_PUBLIC_URL="http://traces.amp.localhost:11080"
# Bare hostnames (no scheme/port) — used as the HTTPRoute hostnames on the plane gateways
export CONSOLE_PUBLIC_HOST="console.amp.localhost"
export API_PUBLIC_HOST="api.amp.localhost"
export OBS_API_PUBLIC_HOST="traces.amp.localhost"
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"
In k3d, every management endpoint is accessed through a plane gateway using a hostname: Thunder, the Console, and the API via the Control Plane kgateway on port 8080 (thunder.amp.localhost, console.amp.localhost, api.amp.localhost), and the Agent Manager Observer via the Observability Plane kgateway on port 11080 (traces.amp.localhost). These match the chart defaults and the Control Plane / Observability Plane OIDC configuration. No port-forwarding is needed; *.localhost names resolve to the loopback address in browsers and curl.
Step 6: Install Thunder Extension (Identity Provider)
Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder.
helm install amp-thunder-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
--version ${VERSION} \
--namespace ${THUNDER_NS} \
--create-namespace \
--timeout 1800s
kubectl wait --for=condition=Available \
deployment -l app.kubernetes.io/instance=amp-thunder-extension \
-n ${THUNDER_NS} --timeout=300s
No MCP overrides are needed here: thunder.bootstrap.agentManagerMcpBaseUrl and
observerMcpBaseUrl become the OAuth resource-server identifiers Thunder
registers for the /mcp endpoints, and their chart defaults are already the k3d
gateway URLs used throughout this guide. If you deviate from those hostnames,
override both to match agentManagerService.config.serverPublicURL and
amObserver.publicUrl — Thunder answers invalid_target unless an MCP client's
requested resource matches a registered identifier exactly. Both are seeded by
the pre-install bootstrap job only, so a helm upgrade will not re-seed them.
Thunder persists its configuration (including the issuer URL) in a database on first boot. If you need to change THUNDER_PUBLIC_URL after installation, you must uninstall the chart, delete its PVC, and reinstall — a helm upgrade alone will not change the issuer in issued tokens.
kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
| grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"http://thunder.amp.localhost:8080"
Step 7: Install OpenChoreo Control Plane
The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension:
helm install openchoreo-control-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 1.1.1 \
--namespace openchoreo-control-plane \
--create-namespace \
--timeout 600s \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-cp.yaml
kubectl wait --for=condition=Available \
deployment --all -n openchoreo-control-plane --timeout=600s
If the install fails with no endpoints available for service "controller-manager-webhook-service", this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the helm install command.
kubectl wait --for=condition=Available deployment --all \
-n openchoreo-control-plane --timeout=300s
What the values file configures
- Backstage disabled (AMP provides its own console)
- OIDC pointing to AMP Thunder Extension (
thunder.amp.localhost:8080) - OpenChoreo API hostname:
api.openchoreo.localhost - Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled
Patch the service-account entitlement claim (required with Thunder ≥ 0.45)
Thunder 0.45+ puts the client name in the client_id claim (its sub is an opaque UUID), but OpenChoreo 1.1.1 extracts service-account entitlements from sub, and the chart does not expose the setting. Without this patch, every service-to-service call is silently unauthorized — the API returns 200 with empty lists and 403s while all pods look healthy, and the gateway bootstrap later fails with Environment 'default' not found.
patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s
for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
if [ "$claim" = "sub" ]; then
kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \
-p '{"spec":{"entitlement":{"claim":"client_id"}}}'
fi
done
Re-apply this patch after any helm upgrade of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's observer-auth-config in Step 10.
Step 8: Setup Data Plane
Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace:
kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
helm install openchoreo-data-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 1.1.1 \
--namespace openchoreo-data-plane \
--create-namespace \
--timeout 600s \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-dp.yaml
kubectl wait --for=condition=Available \
deployment --all -n openchoreo-data-plane --timeout=600s
Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is:
CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
clusterAgent:
clientCA:
value: |
$(echo "$CA_CERT" | sed 's/^/ /')
gateway:
ingress:
external:
name: gateway-default
namespace: openchoreo-data-plane
http:
host: "openchoreoapis.localhost"
listenerName: http
port: 19080
https:
host: "openchoreoapis.localhost"
listenerName: https
port: 19443
secretStoreRef:
name: default
EOF
Step 9: Setup Workflow Plane
Copy the CA certificate:
kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
Install the container registry and the Workflow Plane:
helm upgrade --install registry docker-registry \
--repo https://twuni.github.io/docker-registry.helm \
--namespace openchoreo-workflow-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/v1.1.1/install/k3d/single-cluster/values-registry.yaml \
--timeout 120s
helm install openchoreo-workflow-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
--version 1.1.1 \
--namespace openchoreo-workflow-plane \
--create-namespace \
--timeout 600s
Register the Workflow Plane:
BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
secretStoreRef:
name: default
clusterAgent:
clientCA:
value: |
$(echo "$BP_CA_CERT" | sed 's/^/ /')
EOF
kubectl wait --for=condition=Available \
deployment --all -n openchoreo-workflow-plane --timeout=600s
Step 10: Setup Observability Plane
Copy the CA certificate:
kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
Create the ExternalSecrets for OpenSearch and Observer credentials:
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: opensearch-admin-credentials
namespace: openchoreo-observability-plane
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: default
target:
name: opensearch-admin-credentials
data:
- secretKey: username
remoteRef:
key: opensearch-username
property: value
- secretKey: password
remoteRef:
key: opensearch-password
property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: observer-secret
namespace: openchoreo-observability-plane
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: default
target:
name: observer-secret
data:
- secretKey: OPENSEARCH_USERNAME
remoteRef:
key: opensearch-username
property: value
- secretKey: OPENSEARCH_PASSWORD
remoteRef:
key: opensearch-password
property: value
- secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
remoteRef:
key: observer-oauth-client-secret
property: value
EOF
Wait for the ExternalSecrets to sync:
kubectl wait -n openchoreo-observability-plane \
--for=condition=Ready externalsecret/opensearch-admin-credentials \
externalsecret/observer-secret --timeout=60s
Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/oc-collector-configmap.yaml \
-n openchoreo-observability-plane
Install the Observability Plane:
helm install openchoreo-observability-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
--version 1.1.1 \
--namespace openchoreo-observability-plane \
--create-namespace \
--timeout 25m \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-op.yaml
kubectl wait --for=condition=Available \
deployment --all -n openchoreo-observability-plane --timeout=300s
With Thunder ≥ 0.45, apply the same entitlement-claim patch used for the Control Plane to the observer (without it, log and trace queries return 403 Access denied):
patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s
Install observability modules (logs, metrics, tracing):
# Logs module
helm upgrade --install observability-logs-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
--set adapter.openSearchSecretName="opensearch-admin-credentials" \
--timeout 10m
# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--reuse-values \
--set fluent-bit.enabled=true \
--timeout 10m
# Metrics module
helm upgrade --install observability-metrics-prometheus \
oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.6.1 \
--timeout 10m
# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--set openSearch.enabled=false \
--set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
--set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
--timeout 10m
Register the Observability Plane and link it to other planes:
OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
clusterAgent:
clientCA:
value: |
$(echo "$OP_CA_CERT" | sed 's/^/ /')
observerURL: http://observer.openchoreo.localhost:11080
EOF
# Link Data Plane to Observability Plane
kubectl patch clusterdataplane default -n default --type merge \
-p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
# Link Workflow Plane to Observability Plane
kubectl patch clusterworkflowplane default -n default --type merge \
-p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
Step 11: Verify OpenChoreo Installation
Before proceeding to Phase 2, confirm all planes and Thunder are running:
echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default
All pods should be in Running or Completed state.
Phase 2: Agent Manager Installation
With OpenChoreo running, you can now install the Agent Manager components — the API, console, identity provider, and extensions that provide the AI agent management capabilities.
The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:
- Agent Manager Core : Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
- Extensions : Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.
Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables THUNDER_PUBLIC_URL, THUNDER_INTERNAL_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URL, CONSOLE_PUBLIC_HOST, API_PUBLIC_HOST, OBS_API_PUBLIC_HOST, and INSTRUMENTATION_URL must be set from the Configuration Variables section. The production steps below additionally use AGENTS_DOMAIN — the domain deployed agents are served on, agents.<your base domain> — to build the gateway's hostname and virtual host. The *_PUBLIC_HOST variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.
Core Components
Install these in order — each depends on the one before it.
Step 1: Gateway Operator
Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.
The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator:
kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f -
openssl rand 32 > gateway-aesgcm.key
kubectl create secret generic gateway-encryption-keys \
--namespace ${DATA_PLANE_NS} \
--from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key
rm -f gateway-aesgcm.key
The controller looks for a specific path, /app/data/aesgcm-keys/default-aesgcm256-v1.bin, so the Secret key must be exactly default-aesgcm256-v1.bin. Without it the controller crash-loops on failed to initialize key manager: encryption key file not found for version aesgcm256-v1, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked.
This applies whatever developmentMode is set to. Earlier gateway releases auto-generated a key in development mode; from 1.2.0-beta they do not.
Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted.
helm install gateway-operator \
oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
--version 0.10.1 \
--namespace ${DATA_PLANE_NS} \
--set logging.level=debug \
--set gatewayApi.installStandardCRDs=false \
--set gateway.helm.chartVersion=1.2.0-beta \
--set gateway.values.gateway.controller.encryptionKeys.enabled=true \
--set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \
--timeout 600s
gateway.helm.chartVersion decides which gateway chart the operator deploys, and it is the only thing that decides it — the APIGateway resource carries no chart version. Operator 0.10.1 defaults to 1.2.0-alpha, whose templates predate the controller reading its control-plane address from configuration while still shipping 1.2.0-beta images. The result is a gateway that installs, programs, and serves traffic while never registering with Agent Manager, so it never appears in the gateway list. Pin 1.2.0-beta as above so the chart and the images match.
gateway.values is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired.
Wait for the operator to be ready:
kubectl wait --for=condition=Available \
deployment -l app.kubernetes.io/name=gateway-operator \
-n ${DATA_PLANE_NS} --timeout=300s
Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: wso2-api-platform-gateway-module
rules:
- apiGroups: ["gateway.api-platform.wso2.com"]
resources: ["restapis", "apigateways"]
verbs: ["*"]
- apiGroups: ["gateway.kgateway.dev"]
resources: ["backends"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: wso2-api-platform-gateway-module
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: wso2-api-platform-gateway-module
subjects:
- kind: ServiceAccount
name: cluster-agent-dataplane
namespace: ${DATA_PLANE_NS}
EOF
The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.
Step 2: Agent Manager (API + Console + PostgreSQL)
The core platform: a Go API server, a React web console, and a database.
helm install amp \
oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
--version ${VERSION} \
--namespace ${AMP_NS} \
--create-namespace \
--set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
--set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
--set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
--set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
--set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
--set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
--set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
--set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
--set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
--set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
--set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
--set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
--set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
--timeout 1800s
Wait for all components:
# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s
# API server
kubectl wait --for=condition=Available \
deployment/amp-api -n ${AMP_NS} --timeout=600s
# Console
kubectl wait --for=condition=Available \
deployment/amp-console -n ${AMP_NS} --timeout=600s
keyManager.issuer must be the public Thunder URL — the chart default is the k3d
hostname. The authorization server the API advertises in its RFC 9728 protected
resource metadata is derived from it, so
agentManagerService.config.oauthAuthorizationServers only needs setting if it
must differ. Likewise the MCP audience entry is derived from
serverPublicURL above, so keyManager.audience only needs setting to change
the accepted client IDs.
kubectl get pods -n ${AMP_NS}
# Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus
# amp-postgresql-0 (Running) only when using the in-cluster database
Step 3: Agent Sandbox Module
Agents run as sandboxed pods managed by the Agent Sandbox controller (SandboxTemplate / SandboxWarmPool resources) instead of plain Deployments. This module is required — without it, agent deployments cannot be rendered.
helm upgrade --install agent-sandbox \
oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
--version 0.1.1 \
--namespace ${DATA_PLANE_NS} \
--create-namespace \
--wait \
--timeout 10m \
--set namespace=openchoreo-control-plane \
--set dataPlaneNamespace=${DATA_PLANE_NS} \
--set dataPlaneServiceAccount=cluster-agent-dataplane \
--set upstream.version=v0.4.6
Wait for the controller:
kubectl wait -n agent-sandbox-system \
--for=condition=available \
--timeout=180s \
deployment/agent-sandbox-controller
kubectl get crd \
sandboxtemplates.extensions.agents.x-k8s.io \
sandboxwarmpools.extensions.agents.x-k8s.io \
sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed
kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources
With this module, agents run sandboxed under the standard runc runtime. Optionally, individual environments can run agents under gVisor or Kata Containers for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the gVisor and Kata Containers isolation tier guides.
Step 4: Platform Resources
Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the container registry endpoint used by build workflows to push agent images.
helm install amp-platform-resources \
oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
--version ${VERSION} \
--namespace ${DEFAULT_NS} \
--set apiPlatformGateway.namespace=${DATA_PLANE_NS} \
--timeout 1800s
Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from apiPlatformGateway.namespace. Leaving it empty derives the per-org-env convention <org>-<env>, which is what add-environment.sh creates — but Step 7 installs the gateway extension into ${DATA_PLANE_NS}, matching that chart's own apiGateway.namespace default. Set this to the same namespace, as above.
Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known), so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a Transient error warning in the agent's own log.
If you install the gateway extension into <org>-<env> instead, leave this unset.
Container registry configuration
The chart defaults are configured for a local k3d cluster with an in-cluster registry at host.k3d.internal:10082. For other environments, override the registry settings:
# Example: external registry with a base domain
helm install amp-platform-resources \
oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
--version ${VERSION} \
--namespace ${DEFAULT_NS} \
--set global.baseDomain="yourdomain.com" \
--set global.defaultResources.registry.tlsVerify=true \
--timeout 1800s
# Registry endpoint will be: registry.yourdomain.com
# Example: explicit registry endpoint
helm install amp-platform-resources \
oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
--version ${VERSION} \
--namespace ${DEFAULT_NS} \
--set global.registry.endpoint="your-registry.example.com:5000" \
--set global.defaultResources.registry.tlsVerify=true \
--timeout 1800s| Value | Default | Description |
|---|---|---|
global.registry.endpoint | host.k3d.internal:10082 | Registry endpoint for pushing images |
global.baseDomain | "" | When set, registry endpoint becomes registry.<baseDomain> |
global.defaultResources.registry.tlsVerify | false | Enable TLS verification for registry connections |
Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly:
- Push-to-create. Each build pushes its image as
<workflow-run-name>-image— a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. Amazon ECR does not and fails every build withrepository ... not found; the same applies to any registry that requires repositories to be provisioned first. - Static credentials. The push step reads a fixed
.dockerconfigjsonfrom theregistry-push-secretSecret (or pushes unauthenticated if it is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline.
CNCF Distribution, Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with no authentication; put htpasswd in front of it before it carries anything real.
Extensions
These can be installed in any order after Core is ready.
Step 5: Observability Extension (Agent Manager Observer)
Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.
helm install amp-observability-traces \
oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
--version ${VERSION} \
--namespace ${OBSERVABILITY_NS} \
--set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
--set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
--set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \
--timeout 1800s
kubectl wait --for=condition=Available \
deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
amObserver.auth.issuer must be the public Thunder URL, not the in-cluster
service URL. The observer validates the same user token the console and amctl
send to the Agent Manager API, so its issuer has to match
agentManagerService.config.keyManager.issuer from Step 2. Leave it at the
chart default on a custom-domain install and the traces page stays empty while
the observer logs JWT validation failed ... invalid issuer.
The chart derives amObserver.oauth.authorizationServers and the MCP audience
entry from the values above; set them explicitly only if they must differ.
Step 6: Evaluation Extension
Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.
helm install amp-evaluation-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
--version ${VERSION} \
--namespace ${BUILD_CI_NS} \
--timeout 1800s
Evaluation jobs publish scores using the amp-publisher-client OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (secret/amp-publisher-client-secret) and must match the value set on thunder.bootstrap.ampPublisherClient.clientSecret.
The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the kubernetes Service ClusterIP. Check what that address is:
kubectl -n default get endpoints kubernetes
networkPolicy.evaluationJob.apiServer.cidrs defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet:
helm install amp-evaluation-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
--version ${VERSION} \
--namespace ${BUILD_CI_NS} \
--timeout 1800s \
--set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"
Setting it is required if kubernetes resolves to a public endpoint (public GKE/AKS control planes) or to 100.64.0.0/10. The default will not match those, and every evaluation will publish its scores and then report FAILED.
Step 7: API Platform Gateway Extension
Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. Install this last — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.
helm install api-platform-default-default \
oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
--version ${VERSION} \
--namespace ${DATA_PLANE_NS} \
--set agentManager.orgName=default \
--set gateway.environment=default \
--timeout 1800s
kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
-n ${DATA_PLANE_NS} --timeout=300s
The extension already renders an OTLP ingest route for its own gateway (<release>-otel-restapi in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace <org>-<env>, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace:
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"
kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"
Provision Thunder Identity Provider for the Default Environment
Every Environment needs its own dedicated Thunder ID instance. This is separate from the platform Thunder installed in Phase 1, which only handles console and API login. This instance is what issues each agent its own OAuth2 credential (AgentID) in that Environment. Agents can still be created without it, but they will never get an AgentID there — Agent Manager keeps retrying and failing in the background, and the API Platform Gateway Extension silently skips registering its ThunderKeyManager identity provider, so the Identity Providers page stays empty with no error anywhere in the platform. Run this once the Agent Manager API is reachable at ${API_PUBLIC_URL}:
curl -fsSL "https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment-thunder.sh" \
-o add-environment-thunder.sh
ENV_NAME=default \
DISPLAY_NAME="Default" \
ORG_NAME=default \
AMP_API_URL="${API_PUBLIC_URL}/api/v1" \
IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \
bash add-environment-thunder.sh
kubectl get pods -n amp-thunder-default-default
# All pods should be Running
Verify and Access the Platform
Run a full status check to confirm everything is running:
# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'
| Service | URL |
|---|---|
| Agent Manager Console | http://console.amp.localhost:8080 |
| Agent Manager API | http://api.amp.localhost:8080 |
| Agent Manager Observer | http://traces.amp.localhost:11080 |
| OTLP trace ingest (HTTP) | http://default-default.gateway.localhost:19080/otel |
Default credentials: admin / admin
values-dp.yaml, installed in Step 8, sets gateway.tls.enabled: false even though it already configures httpsPort: 19443 and a certificateRefs entry pointing at the gateway-tls Secret (created earlier by cert-manager). With TLS disabled, the Data Plane's gateway-default Gateway/Service only get an http listener — there is no 19443 listener to connect to, so an HTTPS OTLP endpoint is not reachable on a default install.
Turning gateway.tls.enabled on would not make the OTLP host above reachable over HTTPS anyway: gateway-tls's certificate only covers *.agentmanager.localhost and *.am-gateway.localhost (its configured hostname and SANs), not *.gateway.localhost — the domain default-default.gateway.localhost above actually uses. Reaching HTTPS on that specific host would additionally require reissuing the certificate with *.gateway.localhost in its dnsNames and pointing gateway.tls.hostname at it.
Given that, this setup keeps TLS disabled and uses the HTTP endpoint above for trace ingestion.
Cleanup
Delete the entire k3d cluster and all resources:
k3d cluster delete amp-local
Production Considerations
This installation is designed for development and exploration. For production:
- Identity provider — Replace Thunder dev mode with a proper IdP (Asgardeo, Auth0, Okta)
- JWT authentication — Set
IS_LOCAL_DEV_ENV=falseand configureKEY_MANAGER_JWKS_URLforagent-manager-serviceandagent-manager-observerbefore exposing either service; when unset, JWT signature verification is skipped and only the token payload is decoded - TLS — Replace self-signed certificates with CA-signed certificates
- Secrets backend — Disable OpenBao dev mode; configure persistent storage and proper auth
- Observability storage — Configure persistent volumes for OpenSearch
- Resource sizing — Adjust requests/limits based on workload
- High availability — Deploy multiple replicas of critical components
- Security hardening — Apply network policies, RBAC, pod security standards
Troubleshooting
Pods stuck in Pending
Usually a resource constraint. Check node capacity:
kubectl describe pod <pod-name> -n <namespace>
kubectl top nodes
Increase Colima/Docker Desktop resources if needed.
Gateway not becoming Programmed
kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator
kubectl describe apigateway api-platform-default-default -n openchoreo-data-plane
Plane registration issues
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
OpenSearch connectivity issues
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
Port already in use
Find the process occupying the port and stop it:
lsof -i :<port>
Reference: Configuration Files
All Agent Manager-specific configuration files used in this guide:
| File | Purpose |
|---|---|
| k3d-config.yaml | k3d cluster with all required port mappings |
| coredns-amp-custom.yaml | CoreDNS rewrites for *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, *.gateway.localhost |
| add-environment-thunder.sh | Provisions the per-environment Thunder identity provider |
| values-cp.yaml | Control Plane — Backstage disabled, AMP Thunder OIDC |
| values-dp.yaml | Data Plane — gateway ports, Fluent Bit config |
| values-op.yaml | Observability Plane — standalone OpenSearch, AMP Thunder OIDC |
| values-openbao.yaml | OpenBao — dev mode, Kubernetes auth, pre-seeded secrets |
| oc-collector-configmap.yaml | Custom OTel Collector ConfigMap for trace ingestion |
| wso2-amp-api-platform-gateway-extension | API Platform Gateway Helm chart (includes gateway config, bootstrap, CR) |
| otel-collector-rest-api.yaml | OTel Collector REST API resource |