# On k3d (Locally)

Install the Agent Manager on a local Kubernetes cluster using [k3d](https://k3d.io/) — a lightweight tool that runs Kubernetes inside Docker.

Just want it running?

The [Quick Start Guide](/agent-manager/docs/v1.0.0-alpha1/getting-started/quick-start/.md) installs everything in a single command using a dev container. Use this page when you want to understand the full setup or need to customize it.

Stronger agent isolation tiers

Agents run sandboxed under the standard **runc** runtime by default. Agent Manager also supports stronger per-environment isolation tiers — **gVisor** (userspace kernel) and **Kata Containers** (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the [gVisor](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/kata/.md) setup guides.

## What You Will Get[​](#what-you-will-get "Direct link to What You Will Get")

Agent Manager is a two-layer system installed in two phases:

* **Phase 1 — OpenChoreo (base layer):** [OpenChoreo](https://openchoreo.dev) is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a **Control Plane** for API and configuration, a **Data Plane** for running workloads and gateways, a **Workflow Plane** for builds and CI pipelines, and an **Observability Plane** for traces, logs, and metrics via OpenSearch.

* **Phase 2 — Agent Manager :** The AI agent management platform installed on top of OpenChoreo. It includes **Thunder** (identity provider — installed with OpenChoreo since it provides JWT validation for all planes), the **Console** (web UI), **AMP API** (backend), **AI Gateway**, **PostgreSQL** (database), **Secrets Extension** (OpenBao for runtime secret injection), **Agent Manager Observer** (traces, logs, and metrics), and **Evaluation Engine** (automated agent evaluations).

This guide installs both layers on a single-node k3d cluster on your local machine.

info

This setup is for **development and exploration**. For production deployments, see the [Production Considerations](#production-considerations) section.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### Hardware[​](#hardware "Direct link to Hardware")

| Resource | Minimum      |
| -------- | ------------ |
| RAM      | 8 GB         |
| CPU      | 4 cores      |
| Disk     | \~10 GB free |

### Required Tools[​](#required-tools "Direct link to Required Tools")

| Tool                                               | Version | Purpose                    |
| -------------------------------------------------- | ------- | -------------------------- |
| [Docker](https://docs.docker.com/get-docker/)      | v26.0+  | Container runtime          |
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+  | Kubernetes CLI             |
| [Helm](https://helm.sh/docs/intro/install/)        | v3.12+  | Kubernetes package manager |
| [k3d](https://k3d.io/#installation)                | v5.8+   | Local Kubernetes clusters  |

Verify all tools:

```
docker --version && kubectl version --client && helm version && k3d version
```

macOS with Colima

If you use Colima instead of Docker Desktop, start a dedicated `agent-manager` profile so the install does not interfere with your default Colima instance:

```
colima start --profile agent-manager \
  --vm-type=vz --vz-rosetta --network-address \
  --cpu 4 --memory 8
```

`--network-address` assigns the VM a stable IP so the cluster keeps working after network changes.

Prefix the cluster creation command in Step 1 with `K3D_FIX_DNS=0`.

### Required Ports[​](#required-ports "Direct link to Required Ports")

The following host ports must be free before installation. The Agent Manager services do not need host ports of their own: the Console and API are served through the Control Plane gateway (8080), the Agent Manager Observer through the Observability Plane gateway (11080), and trace ingestion through the Data Plane gateway (19080).

| Port  | Purpose                                                                                                 |
| ----- | ------------------------------------------------------------------------------------------------------- |
| 6550  | Kubernetes API                                                                                          |
| 8080  | Control Plane Gateway (HTTP) — Console, API, Thunder, OpenChoreo API, external AI gateway control plane |
| 8443  | Control Plane Gateway (HTTPS)                                                                           |
| 19080 | Data Plane Gateway (HTTP) — deployed agents, OTLP `/otel` ingest                                        |
| 19443 | Data Plane Gateway (HTTPS)                                                                              |
| 10082 | Container Registry (Workflow Plane)                                                                     |
| 11080 | Observability Plane Gateway (HTTP) — Agent Manager Observer, Observer API                               |
| 11082 | OpenSearch API                                                                                          |
| 11085 | OpenSearch HTTPS                                                                                        |

***

## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform")

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

* **Control Plane** — API server and configuration management for the platform
* **Data Plane** — runs deployed workloads and API gateways
* **Workflow Plane** — builds and CI pipelines for agent deployments
* **Observability Plane** — trace, log, and metrics collection via OpenSearch

This phase installs all four with Agent Manager-specific configuration. **Estimated time: \~15-20 minutes.**

### Step 1: Create k3d Cluster[​](#step-1-create-k3d-cluster "Direct link to Step 1: Create k3d Cluster")

Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings:

```
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/quick-start/k3d-config.yaml \
  | k3d cluster create --config=-
```

Set the kubectl context:

```
k3d kubeconfig merge amp-local --kubeconfig-merge-default
kubectl config use-context k3d-amp-local
```

Generate machine IDs for Fluent Bit log collection:

```
for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do
  docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"
done
```

Verify

```
kubectl cluster-info --context k3d-amp-local
# Should show cluster running at https://0.0.0.0:6550
```

### Step 2: Apply CoreDNS Configuration[​](#step-2-apply-coredns-configuration "Direct link to Step 2: Apply CoreDNS Configuration")

Enables `*.openchoreo.localhost` DNS resolution inside the cluster:

```
kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/v1.1.1/install/k3d/common/coredns-custom.yaml
```

### Step 3: Install Cluster Prerequisites[​](#step-3-install-cluster-prerequisites "Direct link to Step 3: Install Cluster Prerequisites")

These are infrastructure components that OpenChoreo depends on. You only install them once per cluster.

**Gateway API CRDs** — standard Kubernetes resources for managing network gateways and routing:

```
kubectl apply --server-side \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

**cert-manager (v1.19.2)** — automates TLS certificate issuance and renewal:

```
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --wait --timeout 180s
```

**External Secrets Operator (v1.3.2)** — syncs secrets from external stores (like OpenBao) into Kubernetes:

```
helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s
```

**kgateway (v2.2.1)** — the network gateway for OpenChoreo planes:

```
helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
```

### Step 4: Setup Secrets Store (OpenBao)[​](#step-4-setup-secrets-store-openbao "Direct link to Step 4: Setup Secrets Store (OpenBao)")

OpenBao provides secret management for the Workflow Plane and deployed agents:

```
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-openbao.yaml \
  --timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
```

Configure the External Secrets ClusterSecretStore:

```
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF
```

### Step 5: Configuration Variables[​](#step-5-configuration-variables "Direct link to Step 5: Configuration Variables")

Set these variables before continuing. They are used by the remaining Phase 1 steps and all of Phase 2.

```
export VERSION="1.0.0-alpha1"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export THUNDER_NS="amp-thunder"
export OPENCHOREO_INTERNAL_URL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080"
export THUNDER_PUBLIC_URL="http://thunder.amp.localhost:8080"
export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090"
export CONSOLE_PUBLIC_URL="http://console.amp.localhost:8080"
export API_PUBLIC_URL="http://api.amp.localhost:8080"
export OBS_API_PUBLIC_URL="http://traces.amp.localhost:11080"
# Bare hostnames (no scheme/port) — used as the HTTPRoute hostnames on the plane gateways
export CONSOLE_PUBLIC_HOST="console.amp.localhost"
export API_PUBLIC_HOST="api.amp.localhost"
export OBS_API_PUBLIC_HOST="traces.amp.localhost"
export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel"
```

info

In k3d, every management endpoint is accessed through a plane gateway using a hostname: Thunder, the Console, and the API via the Control Plane kgateway on port 8080 (`thunder.amp.localhost`, `console.amp.localhost`, `api.amp.localhost`), and the Agent Manager Observer via the Observability Plane kgateway on port 11080 (`traces.amp.localhost`). These match the chart defaults and the Control Plane / Observability Plane OIDC configuration. No port-forwarding is needed; `*.localhost` names resolve to the loopback address in browsers and curl.

### Step 6: Install Thunder Extension (Identity Provider)[​](#step-6-install-thunder-extension-identity-provider "Direct link to Step 6: Install Thunder Extension (Identity Provider)")

Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder.

```
helm install amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/instance=amp-thunder-extension \
  -n ${THUNDER_NS} --timeout=300s
```

warning

Thunder persists its configuration (including the issuer URL) in a database on first boot. If you need to change `THUNDER_PUBLIC_URL` after installation, you must **uninstall the chart, delete its PVC, and reinstall** — a `helm upgrade` alone will not change the issuer in issued tokens.

Verify

```
kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \
  wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \
  | grep -o '"issuer":"[^"]*"'
# Expected: "issuer":"http://thunder.amp.localhost:8080"
```

### Step 7: Install OpenChoreo Control Plane[​](#step-7-install-openchoreo-control-plane "Direct link to Step 7: Install OpenChoreo Control Plane")

The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension:

```
helm install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.1.1 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --timeout 600s \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-cp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=600s
```

Webhook race condition

If the install fails with `no endpoints available for service "controller-manager-webhook-service"`, this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the `helm install` command.

```
kubectl wait --for=condition=Available deployment --all \
  -n openchoreo-control-plane --timeout=300s
```

What the values file configures

* Backstage disabled (AMP provides its own console)
* OIDC pointing to AMP Thunder Extension (`thunder.amp.localhost:8080`)
* OpenChoreo API hostname: `api.openchoreo.localhost`
* Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled

### Step 8: Setup Data Plane[​](#step-8-setup-data-plane "Direct link to Step 8: Setup Data Plane")

Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace:

```
kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
```

```
helm install openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.1.1 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --timeout 600s \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-dp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s
```

Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is:

```
CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "openchoreoapis.localhost"
          listenerName: http
          port: 19080
        https:
          host: "openchoreoapis.localhost"
          listenerName: https
          port: 19443
  secretStoreRef:
    name: default
EOF
```

### Step 9: Setup Workflow Plane[​](#step-9-setup-workflow-plane "Direct link to Step 9: Setup Workflow Plane")

Copy the CA certificate:

```
kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
```

Install the container registry and the Workflow Plane:

```
helm upgrade --install registry docker-registry \
  --repo https://twuni.github.io/docker-registry.helm \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --values https://raw.githubusercontent.com/openchoreo/openchoreo/v1.1.1/install/k3d/single-cluster/values-registry.yaml \
  --timeout 120s

helm install openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.1.1 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --timeout 600s
```

Register the Workflow Plane:

```
BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  secretStoreRef:
    name: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s
```

### Step 10: Setup Observability Plane[​](#step-10-setup-observability-plane "Direct link to Step 10: Setup Observability Plane")

Copy the CA certificate:

```
kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the ExternalSecrets for OpenSearch and Observer credentials:

```
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF
```

Wait for the ExternalSecrets to sync:

```
kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s
```

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/values/oc-collector-configmap.yaml \
  -n openchoreo-observability-plane
```

Install the Observability Plane:

```
helm install openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.1.1 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --timeout 25m \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-op.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=300s
```

Install observability modules (logs, metrics, tracing):

```
# Logs module
helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set adapter.openSearchSecretName="opensearch-admin-credentials" \
  --timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.6.1 \
  --timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.4.1 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m
```

Register the Observability Plane and link it to other planes:

```
OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo.localhost:11080
EOF

# Link Data Plane to Observability Plane
kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability Plane
kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
```

### Step 11: Verify OpenChoreo Installation[​](#step-11-verify-openchoreo-installation "Direct link to Step 11: Verify OpenChoreo Installation")

Before proceeding to Phase 2, confirm all planes and Thunder are running:

```
echo "--- Thunder ---"
kubectl get pods -n amp-thunder
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default
```

All pods should be in `Running` or `Completed` state.

***

## Phase 2: Agent Manager Installation[​](#phase-2-agent-manager-installation "Direct link to Phase 2: Agent Manager Installation")

With OpenChoreo running, you can now install the Agent Manager components — the API, console, identity provider, and extensions that provide the AI agent management capabilities.

<!-- -->

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

1. **Agent Manager Core :** Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
2. **Extensions :** Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension.

Prerequisites

Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables `THUNDER_PUBLIC_URL`, `THUNDER_INTERNAL_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`, `CONSOLE_PUBLIC_HOST`, `API_PUBLIC_HOST`, `OBS_API_PUBLIC_HOST`, and `INSTRUMENTATION_URL` must be set from the Configuration Variables section. The `*_PUBLIC_HOST` variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways.

***

### Core Components[​](#core-components "Direct link to Core Components")

Install these in order — each depends on the one before it.

#### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator")

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

```
helm install gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.6.0 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gatewayApi.installStandardCRDs=false \
  --set gateway.helm.chartVersion=1.1.1 \
  --timeout 600s
```

Wait for the operator to be ready:

```
kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s
```

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

```
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF
```

info

The API Platform Gateway is deployed as an extension after Agent Manager is running — see Step 7 below.

#### Step 2: Agent Manager (API + Console + PostgreSQL)[​](#step-2-agent-manager-api--console--postgresql "Direct link to Step 2: Agent Manager (API + Console + PostgreSQL)")

The core platform: a Go API server, a React web console, and a PostgreSQL database.

```
helm install amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --set console.config.auth.baseUrl="${THUNDER_PUBLIC_URL}" \
  --set console.config.auth.signInRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.auth.signOutRedirectURL="${CONSOLE_PUBLIC_URL}/login" \
  --set console.config.apiBaseUrl="${API_PUBLIC_URL}" \
  --set agentManagerService.config.amObserverPublicURL="${OBS_API_PUBLIC_URL}" \
  --set console.ocIngress.hostname="${CONSOLE_PUBLIC_HOST}" \
  --set agentManagerService.ocIngress.hostname="${API_PUBLIC_HOST}" \
  --set agentManagerService.config.serverPublicURL="${API_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.issuer="${THUNDER_PUBLIC_URL}" \
  --set agentManagerService.config.keyManager.jwksUrl="${THUNDER_INTERNAL_URL}/oauth2/jwks" \
  --set agentManagerService.config.oidc.tokenUrl="${THUNDER_INTERNAL_URL}/oauth2/token" \
  --set agentManagerService.config.openChoreo.baseURL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" \
  --timeout 1800s
```

Wait for all components:

```
# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s
```

Verify

```
kubectl get pods -n ${AMP_NS}
# Expected: amp-postgresql-0 (Running), amp-api-xxx (Running), amp-console-xxx (Running)
```

#### Step 3: Agent Sandbox Module[​](#step-3-agent-sandbox-module "Direct link to Step 3: Agent Sandbox Module")

Agents run as **sandboxed pods** managed by the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io/) controller (`SandboxTemplate` / `SandboxWarmPool` resources) instead of plain Deployments. This module is **required** — without it, agent deployments cannot be rendered.

```
helm upgrade --install agent-sandbox \
  oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \
  --version 0.1.1 \
  --namespace ${DATA_PLANE_NS} \
  --create-namespace \
  --wait \
  --timeout 10m \
  --set namespace=openchoreo-control-plane \
  --set dataPlaneNamespace=${DATA_PLANE_NS} \
  --set dataPlaneServiceAccount=cluster-agent-dataplane \
  --set upstream.version=v0.4.6
```

Wait for the controller:

```
kubectl wait -n agent-sandbox-system \
  --for=condition=available \
  --timeout=180s \
  deployment/agent-sandbox-controller
```

Verify

```
kubectl get crd \
  sandboxtemplates.extensions.agents.x-k8s.io \
  sandboxwarmpools.extensions.agents.x-k8s.io \
  sandboxclaims.extensions.agents.x-k8s.io
# All three CRDs should be listed

kubectl get clusterrole openchoreo-agent-sandbox-access
# RBAC granting the data-plane agent access to sandbox resources
```

Stronger isolation tiers

With this module, agents run sandboxed under the standard **runc** runtime. Optionally, individual environments can run agents under **gVisor** or **Kata Containers** for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the [gVisor](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/v1.0.0-alpha1/administration/isolation-tiers/kata/.md) isolation tier guides.

#### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources")

Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the **container registry endpoint** used by build workflows to push agent images.

```
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --timeout 1800s
```

<!-- -->

Container registry configuration

The chart defaults are configured for a local k3d cluster with an in-cluster registry at `host.k3d.internal:10082`. For other environments, override the registry settings:

```bash
# Example: external registry with a base domain
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.baseDomain="yourdomain.com" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
# Registry endpoint will be: registry.yourdomain.com

# Example: explicit registry endpoint
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --set global.registry.endpoint="your-registry.example.com:5000" \
  --set global.defaultResources.registry.tlsVerify=true \
  --timeout 1800s
```

| Value                                        | Default                   | Description                                                 |
| -------------------------------------------- | ------------------------- | ----------------------------------------------------------- |
| `global.registry.endpoint`                   | `host.k3d.internal:10082` | Registry endpoint for pushing images                        |
| `global.baseDomain`                          | `""`                      | When set, registry endpoint becomes `registry.<baseDomain>` |
| `global.defaultResources.registry.tlsVerify` | `false`                   | Enable TLS verification for registry connections            |

***

### Extensions[​](#extensions "Direct link to Extensions")

These can be installed in any order after Core is ready.

#### Step 5: Observability Extension (Agent Manager Observer)[​](#step-5-observability-extension-agent-manager-observer "Direct link to Step 5: Observability Extension (Agent Manager Observer)")

Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI.

```
helm install amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \
  --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s
```

#### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension")

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

```
helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s
```

info

The default `publisher.apiKey` must match `publisherApiKey.value` in the Agent Manager chart. Both default to `amp-internal-api-key`.

#### Step 7: API Platform Gateway Extension[​](#step-7-api-platform-gateway-extension "Direct link to Step 7: API Platform Gateway Extension")

Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

```
helm install api-platform-default-default \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set agentManager.orgName=default \
  --set gateway.environment=default \
  --timeout 1800s

kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s
```

After the gateway is running, apply the OTEL trace collection RestApi:

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml
```

Verify

```
kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"

kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap
# STATUS should show "Complete"
```

***

## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform")

Run a full status check to confirm everything is running:

```
# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'
```

| Service                       | URL                                                    |
| ----------------------------- | ------------------------------------------------------ |
| **Agent Manager Console**     | <http://console.amp.localhost:8080>                    |
| **Agent Manager API**         | <http://api.amp.localhost:8080>                        |
| **Agent Manager Observer**    | <http://traces.amp.localhost:11080>                    |
| **OTLP trace ingest (HTTP)**  | <http://default-default.gateway.localhost:19080/otel>  |
| **OTLP trace ingest (HTTPS)** | <https://default-default.gateway.localhost:19443/otel> |

**Default credentials:** `admin` / `admin`

### Self-Signed Certificate for OTEL HTTPS[​](#self-signed-certificate-for-otel-https "Direct link to Self-Signed Certificate for OTEL HTTPS")

If your OTEL exporters encounter certificate errors on the HTTPS ingest endpoint (the Data Plane gateway serves a self-signed certificate by default):

```
kubectl get secret gateway-tls \
  -n openchoreo-data-plane \
  -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt

export OTEL_EXPORTER_OTLP_CERTIFICATE=$(pwd)/ca.crt
```

***

## Cleanup[​](#cleanup "Direct link to Cleanup")

Delete the entire k3d cluster and all resources:

```
k3d cluster delete amp-local
```

***

## Production Considerations[​](#production-considerations "Direct link to Production Considerations")

This installation is designed for development and exploration. For production:

1. **Identity provider** — Replace Thunder dev mode with a proper IdP (Asgardeo, Auth0, Okta)
2. **TLS** — Replace self-signed certificates with CA-signed certificates
3. **Secrets backend** — Disable OpenBao dev mode; configure persistent storage and proper auth
4. **Observability storage** — Configure persistent volumes for OpenSearch
5. **Resource sizing** — Adjust requests/limits based on workload
6. **High availability** — Deploy multiple replicas of critical components
7. **Security hardening** — Apply network policies, RBAC, pod security standards

***

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Pods stuck in Pending

Usually a resource constraint. Check node capacity:

```
kubectl describe pod <pod-name> -n <namespace>
kubectl top nodes
```

Increase Colima/Docker Desktop resources if needed.

Gateway not becoming Programmed

```
kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator
kubectl describe apigateway api-platform-default-default -n openchoreo-data-plane
```

Plane registration issues

```
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
```

OpenSearch connectivity issues

```
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
```

Port already in use

Find the process occupying the port and stop it:

```
lsof -i :<port>
```

***

## Reference: Configuration Files[​](#reference-configuration-files "Direct link to Reference: Configuration Files")

All Agent Manager-specific configuration files used in this guide:

| File                                                                                                                                                      | Purpose                                                                  |
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| [k3d-config.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/quick-start/k3d-config.yaml)                         | k3d cluster with all required port mappings                              |
| [values-cp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-cp.yaml)                        | Control Plane — Backstage disabled, AMP Thunder OIDC                     |
| [values-dp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-dp.yaml)                        | Data Plane — gateway ports, Fluent Bit config                            |
| [values-op.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-op.yaml)                        | Observability Plane — standalone OpenSearch, AMP Thunder OIDC            |
| [values-openbao.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/single-cluster/values-openbao.yaml)              | OpenBao — dev mode, Kubernetes auth, pre-seeded secrets                  |
| [oc-collector-configmap.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/values/oc-collector-configmap.yaml)      | Custom OTel Collector ConfigMap for trace ingestion                      |
| [wso2-amp-api-platform-gateway-extension](https://github.com/wso2/agent-manager/tree/amp/deployments/helm-charts/wso2-amp-api-platform-gateway-extension) | API Platform Gateway Helm chart (includes gateway config, bootstrap, CR) |
| [otel-collector-rest-api.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0-alpha1/deployments/values/otel-collector-rest-api.yaml)    | OTel Collector REST API resource                                         |