Skip to main content
Version: Next

On k3d (Locally)

Install the Agent Manager on a local Kubernetes cluster using k3d — a lightweight tool that runs Kubernetes inside Docker.

Just want it running?

The Quick Start Guide installs everything in a single command using a dev container. Use this page when you want to understand the full setup or need to customize it.

Stronger agent isolation tiers

Agents run sandboxed under the standard runc runtime by default. Agent Manager also supports stronger per-environment isolation tiers — gVisor (userspace kernel) and Kata Containers (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the gVisor and Kata Containers setup guides.

What You Will Get

Agent Manager is a two-layer system installed in two phases:

  • Phase 1 — OpenChoreo (base layer): OpenChoreo is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a Control Plane for API and configuration, a Data Plane for running workloads and gateways, a Workflow Plane for builds and CI pipelines, and an Observability Plane for traces, logs, and metrics via OpenSearch.

  • Phase 2 — Agent Manager : The AI agent management platform installed on top of OpenChoreo. It includes Thunder (identity provider — installed with OpenChoreo since it provides JWT validation for all planes), the Console (web UI), AMP API (backend), AI Gateway, PostgreSQL (database), Secrets Extension (OpenBao for runtime secret injection), Agent Manager Observer (traces, logs, and metrics), and Evaluation Engine (automated agent evaluations).

This guide installs both layers on a single-node k3d cluster on your local machine.

info

This setup is for development and exploration. For production deployments, see the Production Considerations section.

Prerequisites

Hardware

ResourceMinimum
RAM8 GB
CPU4 cores
Disk~10 GB free

Required Tools

ToolVersionPurpose
Dockerv26.0+Container runtime
kubectlv1.32+Kubernetes CLI
Helmv3.12+Kubernetes package manager
k3dv5.8+Local Kubernetes clusters

Verify all tools:

docker --version && kubectl version --client && helm version && k3d version
macOS with Colima

If you use Colima instead of Docker Desktop, start a dedicated agent-manager profile so the install does not interfere with your default Colima instance:

colima start --profile agent-manager \
--vm-type=vz --vz-rosetta --network-address \
--cpu 4 --memory 8

--network-address assigns the VM a stable IP so the cluster keeps working after network changes.

Prefix the cluster creation command in Step 1 with K3D_FIX_DNS=0.

Required Ports

The following host ports must be free before installation. The Agent Manager services do not need host ports of their own: the Console and API are served through the Control Plane gateway (8080), the Agent Manager Observer through the Observability Plane gateway (11080), and trace ingestion through the Data Plane gateway (19080).

PortPurpose
6550Kubernetes API
8080Control Plane Gateway (HTTP) — Console, API, Thunder, OpenChoreo API, external AI gateway control plane
8443Control Plane Gateway (HTTPS)
19080Data Plane Gateway (HTTP) — deployed agents, OTLP /otel ingest
19443Data Plane Gateway (HTTPS)
10082Container Registry (Workflow Plane)
11080Observability Plane Gateway (HTTP) — Agent Manager Observer, Observer API
11082OpenSearch API
11085OpenSearch HTTPS

Phase 1: OpenChoreo Platform

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

  • Control Plane — API server and configuration management for the platform
  • Data Plane — runs deployed workloads and API gateways
  • Workflow Plane — builds and CI pipelines for agent deployments
  • Observability Plane — trace, log, and metrics collection via OpenSearch

This phase installs all four with Agent Manager-specific configuration. Estimated time: ~15-20 minutes.

Step 1: Create k3d Cluster

Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings:

curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/quick-start/k3d-config.yaml \
| k3d cluster create --config=-

Set the kubectl context:

k3d kubeconfig merge amp-local --kubeconfig-merge-default
kubectl config use-context k3d-amp-local

Generate machine IDs for Fluent Bit log collection:

for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do
docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"
done
Verify
kubectl cluster-info --context k3d-amp-local
# Should show cluster running at https://0.0.0.0:6550

Step 2: Apply CoreDNS Configuration

Enables *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, and *.gateway.localhost DNS resolution inside the cluster — required for in-cluster lookups of the Console/API/Thunder hostnames as well as agent-to-agent and AI gateway traffic:

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/k8s/coredns-amp-custom.yaml

CoreDNS's reload plugin can miss the override if the ConfigMap mounts after the pod's initial parse, so restart it to make sure the rewrite rules are active before anything depends on them:

kubectl rollout restart deployment/coredns -n kube-system
kubectl rollout status deployment/coredns -n kube-system --timeout=60s

Step 3: Install Cluster Prerequisites

These are infrastructure components that OpenChoreo depends on. You only install them once per cluster.

Gateway API CRDs — standard Kubernetes resources for managing network gateways and routing:

kubectl apply --server-side \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

cert-manager (v1.19.2) — automates TLS certificate issuance and renewal:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.19.2 \
--set crds.enabled=true \
--wait --timeout 180s

External Secrets Operator (v1.3.2) — syncs secrets from external stores (like OpenBao) into Kubernetes:

helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
--namespace external-secrets \
--create-namespace \
--version 1.3.2 \
--set installCRDs=true \
--wait --timeout 180s

kgateway (v2.2.1) — the network gateway for OpenChoreo planes:

helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--create-namespace \
--namespace openchoreo-control-plane \
--version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
--namespace openchoreo-control-plane \
--create-namespace \
--version v2.2.1 \
--set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true

Step 4: Setup Secrets Store (OpenBao)

OpenBao provides secret management for the Workflow Plane and deployed agents:

helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
--namespace openbao \
--create-namespace \
--version 0.25.6 \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-openbao.yaml \
--timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s

Configure the External Secrets ClusterSecretStore:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-secrets-openbao
namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: default
spec:
provider:
vault:
server: "http://openbao.openbao.svc:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "openchoreo-secret-writer-role"
serviceAccountRef:
name: "external-secrets-openbao"
namespace: "openbao"
EOF

Step 5: Configuration Variables

Set these variables before continuing. They are used by the remaining Phase 1 steps and all of Phase 2.

export VERSION="0.0.0-dev"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"
export OPENCHOREO_INTERNAL_URL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"
export THUNDER_PUBLIC_URL="http://thunder.amp.localhost:8080"
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"
export CONSOLE_PUBLIC_URL="http://console.amp.localhost:8080"
export API_PUBLIC_URL="http://api.amp.localhost:8080"
export OBS_API_PUBLIC_URL="http://traces.amp.localhost:11080"
# Bare hostnames (no scheme/port) — used as the HTTPRoute hostnames on the plane gateways
export CONSOLE_PUBLIC_HOST="console.amp.localhost"
export API_PUBLIC_HOST="api.amp.localhost"
export OBS_API_PUBLIC_HOST="traces.amp.localhost"
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"
info

In k3d, every management endpoint is accessed through a plane gateway using a hostname: Thunder, the Console, and the API via the Control Plane kgateway on port 8080 (thunder.amp.localhost, console.amp.localhost, api.amp.localhost), and the Agent Manager Observer via the Observability Plane kgateway on port 11080 (traces.amp.localhost). These match the chart defaults and the Control Plane / Observability Plane OIDC configuration. No port-forwarding is needed; *.localhost names resolve to the loopback address in browsers and curl.

Step 6: Install Thunder Extension (Identity Provider)

Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder.

helm install amp-thunder-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
--version ${VERSION} \
--namespace ${THUNDER_NS} \
--create-namespace \
--timeout 1800s

kubectl wait --for=condition=Available \
deployment -l app.kubernetes.io/instance=amp-thunder-extension \
-n ${THUNDER_NS} --timeout=300s
info

No MCP overrides are needed here: thunder.bootstrap.agentManagerMcpBaseUrl and observerMcpBaseUrl become the OAuth resource-server identifiers Thunder registers for the /mcp endpoints, and their chart defaults are already the k3d gateway URLs used throughout this guide. If you deviate from those hostnames, override both to match agentManagerService.config.serverPublicURL and amObserver.publicUrl — Thunder answers invalid_target unless an MCP client's requested resource matches a registered identifier exactly. Both are seeded by the pre-install bootstrap job only, so a helm upgrade will not re-seed them.

warning

Thunder persists its configuration (including the issuer URL) in a database on first boot. If you need to change THUNDER_PUBLIC_URL after installation, you must uninstall the chart, delete its PVC, and reinstall — a helm upgrade alone will not change the issuer in issued tokens.

Verify
kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
| grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"http://thunder.amp.localhost:8080"

Step 7: Install OpenChoreo Control Plane

The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension:

helm install openchoreo-control-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
--version 1.1.1 \
--namespace openchoreo-control-plane \
--create-namespace \
--timeout 600s \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-cp.yaml

kubectl wait --for=condition=Available \
deployment --all -n openchoreo-control-plane --timeout=600s
Webhook race condition

If the install fails with no endpoints available for service "controller-manager-webhook-service", this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the helm install command.

kubectl wait --for=condition=Available deployment --all \
-n openchoreo-control-plane --timeout=300s
What the values file configures
  • Backstage disabled (AMP provides its own console)
  • OIDC pointing to AMP Thunder Extension (thunder.amp.localhost:8080)
  • OpenChoreo API hostname: api.openchoreo.localhost
  • Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled

Patch the service-account entitlement claim (required with Thunder ≥ 0.45)

Thunder 0.45+ puts the client name in the client_id claim (its sub is an opaque UUID), but OpenChoreo 1.1.1 extracts service-account entitlements from sub, and the chart does not expose the setting. Without this patch, every service-to-service call is silently unauthorized — the API returns 200 with empty lists and 403s while all pods look healthy, and the gateway bootstrap later fails with Environment 'default' not found.

patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -

kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane
kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s

for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do
claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}')
if [ "$claim" = "sub" ]; then
kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \
-p '{"spec":{"entitlement":{"claim":"client_id"}}}'
fi
done
note

Re-apply this patch after any helm upgrade of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's observer-auth-config in Step 10.

Step 8: Setup Data Plane

Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace:

kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
helm install openchoreo-data-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
--version 1.1.1 \
--namespace openchoreo-data-plane \
--create-namespace \
--timeout 600s \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-dp.yaml

kubectl wait --for=condition=Available \
deployment --all -n openchoreo-data-plane --timeout=600s

Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is:

CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
clusterAgent:
clientCA:
value: |
$(echo "$CA_CERT" | sed 's/^/ /')
gateway:
ingress:
external:
name: gateway-default
namespace: openchoreo-data-plane
http:
host: "openchoreoapis.localhost"
listenerName: http
port: 19080
https:
host: "openchoreoapis.localhost"
listenerName: https
port: 19443
secretStoreRef:
name: default
EOF

Step 9: Setup Workflow Plane

Copy the CA certificate:

kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

Install the container registry and the Workflow Plane:

helm upgrade --install registry docker-registry \
--repo https://twuni.github.io/docker-registry.helm \
--namespace openchoreo-workflow-plane \
--create-namespace \
--values https://raw.githubusercontent.com/openchoreo/openchoreo/v1.1.1/install/k3d/single-cluster/values-registry.yaml \
--timeout 120s

helm install openchoreo-workflow-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
--version 1.1.1 \
--namespace openchoreo-workflow-plane \
--create-namespace \
--timeout 600s

Register the Workflow Plane:

BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
secretStoreRef:
name: default
clusterAgent:
clientCA:
value: |
$(echo "$BP_CA_CERT" | sed 's/^/ /')
EOF

kubectl wait --for=condition=Available \
deployment --all -n openchoreo-workflow-plane --timeout=600s

Step 10: Setup Observability Plane

Copy the CA certificate:

kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
-n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
--from-literal=ca.crt="$CA_CRT" \
-n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

Create the ExternalSecrets for OpenSearch and Observer credentials:

kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: opensearch-admin-credentials
namespace: openchoreo-observability-plane
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: default
target:
name: opensearch-admin-credentials
data:
- secretKey: username
remoteRef:
key: opensearch-username
property: value
- secretKey: password
remoteRef:
key: opensearch-password
property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: observer-secret
namespace: openchoreo-observability-plane
spec:
refreshInterval: 1h
secretStoreRef:
kind: ClusterSecretStore
name: default
target:
name: observer-secret
data:
- secretKey: OPENSEARCH_USERNAME
remoteRef:
key: opensearch-username
property: value
- secretKey: OPENSEARCH_PASSWORD
remoteRef:
key: opensearch-password
property: value
- secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
remoteRef:
key: observer-oauth-client-secret
property: value
EOF

Wait for the ExternalSecrets to sync:

kubectl wait -n openchoreo-observability-plane \
--for=condition=Ready externalsecret/opensearch-admin-credentials \
externalsecret/observer-secret --timeout=60s

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/oc-collector-configmap.yaml \
-n openchoreo-observability-plane

Install the Observability Plane:

helm install openchoreo-observability-plane \
oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
--version 1.1.1 \
--namespace openchoreo-observability-plane \
--create-namespace \
--timeout 25m \
--values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-op.yaml

kubectl wait --for=condition=Available \
deployment --all -n openchoreo-observability-plane --timeout=300s

With Thunder ≥ 0.45, apply the same entitlement-claim patch used for the Control Plane to the observer (without it, log and trace queries return 403 Access denied):

patched_yaml=$(kubectl get configmap observer-auth-config -n openchoreo-observability-plane -o yaml \
| sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g")
echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f -
kubectl rollout restart deployment/observer -n openchoreo-observability-plane
kubectl rollout status deployment/observer -n openchoreo-observability-plane --timeout=120s

Install observability modules (logs, metrics, tracing):

# Logs module
helm upgrade --install observability-logs-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
--set adapter.openSearchSecretName="opensearch-admin-credentials" \
--timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--reuse-values \
--set fluent-bit.enabled=true \
--timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.6.1 \
--timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
--create-namespace \
--namespace openchoreo-observability-plane \
--version 0.4.1 \
--set openSearch.enabled=false \
--set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
--set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
--timeout 10m

Register the Observability Plane and link it to other planes:

OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
-n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
name: default
namespace: default
spec:
planeID: "default"
clusterAgent:
clientCA:
value: |
$(echo "$OP_CA_CERT" | sed 's/^/ /')
observerURL: http://observer.openchoreo.localhost:11080
EOF

# Link Data Plane to Observability Plane
kubectl patch clusterdataplane default -n default --type merge \
-p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability Plane
kubectl patch clusterworkflowplane default -n default --type merge \
-p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

Step 11: Verify OpenChoreo Installation

Before proceeding to Phase 2, confirm all planes and Thunder are running:

echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default

All pods should be in Running or Completed state.


Phase 2: Agent Manager Installation

With OpenChoreo running, you can now install the Agent Manager components — the API, console, identity provider, and extensions that provide the AI agent management capabilities.

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

  1. Agent Manager Core : Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
  2. Extensions : Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.
Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables THUNDER_PUBLIC_URL, THUNDER_INTERNAL_URL, CONSOLE_PUBLIC_URL, API_PUBLIC_URL, OBS_API_PUBLIC_URL, CONSOLE_PUBLIC_HOST, API_PUBLIC_HOST, OBS_API_PUBLIC_HOST, and INSTRUMENTATION_URL must be set from the Configuration Variables section. The production steps below additionally use AGENTS_DOMAIN — the domain deployed agents are served on, agents.<your base domain> — to build the gateway's hostname and virtual host. The *_PUBLIC_HOST variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.


Core Components

Install these in order — each depends on the one before it.

Step 1: Gateway Operator

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator:

kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f -

openssl rand 32 > gateway-aesgcm.key
kubectl create secret generic gateway-encryption-keys \
--namespace ${DATA_PLANE_NS} \
--from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key
rm -f gateway-aesgcm.key
The key is required, and the file name is fixed

The controller looks for a specific path, /app/data/aesgcm-keys/default-aesgcm256-v1.bin, so the Secret key must be exactly default-aesgcm256-v1.bin. Without it the controller crash-loops on failed to initialize key manager: encryption key file not found for version aesgcm256-v1, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked.

This applies whatever developmentMode is set to. Earlier gateway releases auto-generated a key in development mode; from 1.2.0-beta they do not.

Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted.

helm install gateway-operator \
oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
--version 0.10.1 \
--namespace ${DATA_PLANE_NS} \
--set logging.level=debug \
--set gatewayApi.installStandardCRDs=false \
--set gateway.helm.chartVersion=1.2.0-beta \
--set gateway.values.gateway.controller.encryptionKeys.enabled=true \
--set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \
--timeout 600s
Pin the gateway chart, do not inherit it

gateway.helm.chartVersion decides which gateway chart the operator deploys, and it is the only thing that decides it — the APIGateway resource carries no chart version. Operator 0.10.1 defaults to 1.2.0-alpha, whose templates predate the controller reading its control-plane address from configuration while still shipping 1.2.0-beta images. The result is a gateway that installs, programs, and serves traffic while never registering with Agent Manager, so it never appears in the gateway list. Pin 1.2.0-beta as above so the chart and the images match.

gateway.values is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired.

Wait for the operator to be ready:

kubectl wait --for=condition=Available \
deployment -l app.kubernetes.io/name=gateway-operator \
-n ${DATA_PLANE_NS} --timeout=300s

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: wso2-api-platform-gateway-module
rules:
- apiGroups: ["gateway.api-platform.wso2.com"]
resources: ["restapis", "apigateways"]
verbs: ["*"]
- apiGroups: ["gateway.kgateway.dev"]
resources: ["backends"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: wso2-api-platform-gateway-module
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: wso2-api-platform-gateway-module
subjects:
- kind: ServiceAccount
name: cluster-agent-dataplane
namespace: ${DATA_PLANE_NS}
EOF
info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

Step 2: Agent Manager (API + Console + PostgreSQL)

The core platform: a Go API server, a React web console, and a database.

helm install amp \
oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
--version ${VERSION} \
--namespace ${AMP_NS} \
--create-namespace \
--set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
--set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
--set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
--set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
--set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
--set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
--set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
--set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
--set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
--set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
--set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
--set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
--set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
--timeout 1800s

Wait for all components:

# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
deployment/amp-console -n ${AMP_NS} --timeout=600s
info

keyManager.issuer must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so agentManagerService.config.oauthAuthorizationServers only needs setting if it must differ. Likewise the MCP audience entry is derived from serverPublicURL above, so keyManager.audience only needs setting to change the accepted client IDs.

Verify
kubectl get pods -n ${AMP_NS}
# Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus
# amp-postgresql-0 (Running) only when using the in-cluster database

Step 3: Agent Sandbox Module

Agents run as sandboxed pods managed by the Agent Sandbox controller (SandboxTemplate / SandboxWarmPool resources) instead of plain Deployments. This module is required — without it, agent deployments cannot be rendered.

helm upgrade --install agent-sandbox \
oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
--version 0.1.1 \
--namespace ${DATA_PLANE_NS} \
--create-namespace \
--wait \
--timeout 10m \
--set namespace=openchoreo-control-plane \
--set dataPlaneNamespace=${DATA_PLANE_NS} \
--set dataPlaneServiceAccount=cluster-agent-dataplane \
--set upstream.version=v0.4.6

Wait for the controller:

kubectl wait -n agent-sandbox-system \
--for=condition=available \
--timeout=180s \
deployment/agent-sandbox-controller
Verify
kubectl get crd \
sandboxtemplates.extensions.agents.x-k8s.io \
sandboxwarmpools.extensions.agents.x-k8s.io \
sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources
Stronger isolation tiers

With this module, agents run sandboxed under the standard runc runtime. Optionally, individual environments can run agents under gVisor or Kata Containers for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the gVisor and Kata Containers isolation tier guides.

Step 4: Platform Resources

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the container registry endpoint used by build workflows to push agent images.

helm install amp-platform-resources \
oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
--version ${VERSION} \
--namespace ${DEFAULT_NS} \
--set apiPlatformGateway.namespace=${DATA_PLANE_NS} \
--timeout 1800s
Point the traits at the namespace the gateway is actually in

Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from apiPlatformGateway.namespace. Leaving it empty derives the per-org-env convention <org>-<env>, which is what add-environment.sh creates — but Step 7 installs the gateway extension into ${DATA_PLANE_NS}, matching that chart's own apiGateway.namespace default. Set this to the same namespace, as above.

Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known), so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a Transient error warning in the agent's own log.

If you install the gateway extension into <org>-<env> instead, leave this unset.

Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at host.k3d.internal:10082. For other environments, override the registry settings:

# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
ValueDefaultDescription
global.registry.endpointhost.k3d.internal:10082Registry endpoint for pushing images
global.baseDomain""When set, registry endpoint becomes registry.<baseDomain>
global.defaultResources.registry.tlsVerifyfalseEnable TLS verification for registry connections
Not every registry can back build workflows

Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly:

  • Push-to-create. Each build pushes its image as <workflow-run-name>-image — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. Amazon ECR does not and fails every build with repository ... not found; the same applies to any registry that requires repositories to be provisioned first.
  • Static credentials. The push step reads a fixed .dockerconfigjson from the registry-push-secret Secret (or pushes unauthenticated if it is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline.

CNCF Distribution, Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with no authentication; put htpasswd in front of it before it carries anything real.


Extensions

These can be installed in any order after Core is ready.

Step 5: Observability Extension (Agent Manager Observer)

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

helm install amp-observability-traces \
oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
--version ${VERSION} \
--namespace ${OBSERVABILITY_NS} \
--set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
--set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
--set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \
--timeout 1800s

kubectl wait --for=condition=Available \
deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
warning

amObserver.auth.issuer must be the public Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and amctl send to the Agent Manager API, so its issuer has to match agentManagerService.config.keyManager.issuer from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs JWT validation failed ... invalid issuer.

The chart derives amObserver.oauth.authorizationServers and the MCP audience entry from the values above; set them explicitly only if they must differ.

Step 6: Evaluation Extension

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

helm install amp-evaluation-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
--version ${VERSION} \
--namespace ${BUILD_CI_NS} \
--timeout 1800s
info

Evaluation jobs publish scores using the amp-publisher-client OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (secret/amp-publisher-client-secret) and must match the value set on thunder.bootstrap.ampPublisherClient.clientSecret.

caution

The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the kubernetes Service ClusterIP. Check what that address is:

kubectl -n default get endpoints kubernetes

networkPolicy.evaluationJob.apiServer.cidrs defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet:

helm install amp-evaluation-extension \
oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
--version ${VERSION} \
--namespace ${BUILD_CI_NS} \
--timeout 1800s \
--set "networkPolicy.evaluationJob.apiServer.cidrs[0]=<control-plane-subnet>"

Setting it is required if kubernetes resolves to a public endpoint (public GKE/AKS control planes) or to 100.64.0.0/10. The default will not match those, and every evaluation will publish its scores and then report FAILED.

Step 7: API Platform Gateway Extension

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. Install this last — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

helm install api-platform-default-default \
oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
--version ${VERSION} \
--namespace ${DATA_PLANE_NS} \
--set agentManager.orgName=default \
--set gateway.environment=default \
--timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
-n ${DATA_PLANE_NS} --timeout=300s

The extension already renders an OTLP ingest route for its own gateway (<release>-otel-restapi in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace <org>-<env>, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace:

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml
Verify
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"

Provision Thunder Identity Provider for the Default Environment

Every Environment needs its own dedicated Thunder ID instance. This is separate from the platform Thunder installed in Phase 1, which only handles console and API login. This instance is what issues each agent its own OAuth2 credential (AgentID) in that Environment. Agents can still be created without it, but they will never get an AgentID there — Agent Manager keeps retrying and failing in the background, and the API Platform Gateway Extension silently skips registering its ThunderKeyManager identity provider, so the Identity Providers page stays empty with no error anywhere in the platform. Run this once the Agent Manager API is reachable at ${API_PUBLIC_URL}:

curl -fsSL "https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment-thunder.sh" \
-o add-environment-thunder.sh

ENV_NAME=default \
DISPLAY_NAME="Default" \
ORG_NAME=default \
AMP_API_URL="${API_PUBLIC_URL}/api/v1" \
IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \
bash add-environment-thunder.sh
Verify
kubectl get pods -n amp-thunder-default-default
# All pods should be Running

Verify and Access the Platform

Run a full status check to confirm everything is running:

# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'
ServiceURL
Agent Manager Consolehttp://console.amp.localhost:8080
Agent Manager APIhttp://api.amp.localhost:8080
Agent Manager Observerhttp://traces.amp.localhost:11080
OTLP trace ingest (HTTP)http://default-default.gateway.localhost:19080/otel

Default credentials: admin / admin

HTTPS ingest is disabled by default on this setup, and enabling it doesn't cover the documented hosts

values-dp.yaml, installed in Step 8, sets gateway.tls.enabled: false even though it already configures httpsPort: 19443 and a certificateRefs entry pointing at the gateway-tls Secret (created earlier by cert-manager). With TLS disabled, the Data Plane's gateway-default Gateway/Service only get an http listener — there is no 19443 listener to connect to, so an HTTPS OTLP endpoint is not reachable on a default install.

Turning gateway.tls.enabled on would not make the OTLP host above reachable over HTTPS anyway: gateway-tls's certificate only covers *.agentmanager.localhost and *.am-gateway.localhost (its configured hostname and SANs), not *.gateway.localhost — the domain default-default.gateway.localhost above actually uses. Reaching HTTPS on that specific host would additionally require reissuing the certificate with *.gateway.localhost in its dnsNames and pointing gateway.tls.hostname at it.

Given that, this setup keeps TLS disabled and uses the HTTP endpoint above for trace ingestion.


Cleanup

Delete the entire k3d cluster and all resources:

k3d cluster delete amp-local

Production Considerations

This installation is designed for development and exploration. For production:

  1. Identity provider — Replace Thunder dev mode with a proper IdP (Asgardeo, Auth0, Okta)
  2. JWT authentication — Set IS_LOCAL_DEV_ENV=false and configure KEY_MANAGER_JWKS_URL for agent-manager-service and agent-manager-observer before exposing either service; when unset, JWT signature verification is skipped and only the token payload is decoded
  3. TLS — Replace self-signed certificates with CA-signed certificates
  4. Secrets backend — Disable OpenBao dev mode; configure persistent storage and proper auth
  5. Observability storage — Configure persistent volumes for OpenSearch
  6. Resource sizing — Adjust requests/limits based on workload
  7. High availability — Deploy multiple replicas of critical components
  8. Security hardening — Apply network policies, RBAC, pod security standards

Troubleshooting

Pods stuck in Pending

Usually a resource constraint. Check node capacity:

kubectl describe pod <pod-name> -n <namespace>
kubectl top nodes

Increase Colima/Docker Desktop resources if needed.

Gateway not becoming Programmed
kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator
kubectl describe apigateway api-platform-default-default -n openchoreo-data-plane
Plane registration issues
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
OpenSearch connectivity issues
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
Port already in use

Find the process occupying the port and stop it:

lsof -i :<port>

Reference: Configuration Files

All Agent Manager-specific configuration files used in this guide:

FilePurpose
k3d-config.yamlk3d cluster with all required port mappings
coredns-amp-custom.yamlCoreDNS rewrites for *.openchoreo.localhost, *.amp.localhost, *.agentmanager.localhost, *.am-gateway.localhost, *.gateway.localhost
add-environment-thunder.shProvisions the per-environment Thunder identity provider
values-cp.yamlControl Plane — Backstage disabled, AMP Thunder OIDC
values-dp.yamlData Plane — gateway ports, Fluent Bit config
values-op.yamlObservability Plane — standalone OpenSearch, AMP Thunder OIDC
values-openbao.yamlOpenBao — dev mode, Kubernetes auth, pre-seeded secrets
oc-collector-configmap.yamlCustom OTel Collector ConfigMap for trace ingestion
wso2-amp-api-platform-gateway-extensionAPI Platform Gateway Helm chart (includes gateway config, bootstrap, CR)
otel-collector-rest-api.yamlOTel Collector REST API resource