# On k3d (Locally)

Install the Agent Manager on a local Kubernetes cluster using [k3d](https://k3d.io/) — a lightweight tool that runs Kubernetes inside Docker.

Just want it running?

The [Quick Start Guide](/agent-manager/docs/v0.11.x/getting-started/quick-start/.md) installs everything in a single command using a dev container. Use this page when you want to understand the full setup or need to customize it.

## What You Will Get[​](#what-you-will-get "Direct link to What You Will Get")

Agent Manager is a two-layer system installed in two phases:

* **Phase 1 — OpenChoreo (base layer):** [OpenChoreo](https://openchoreo.dev) is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a **Control Plane** for API and configuration, a **Data Plane** for running workloads and gateways, a **Workflow Plane** for builds and CI pipelines, and an **Observability Plane** for traces, logs, and metrics via OpenSearch.

* **Phase 2 — Agent Manager :** The AI agent management platform installed on top of OpenChoreo. It includes the **Console** (web UI), **AMP API** (backend), **Thunder** (identity provider), **AI Gateway**, **PostgreSQL** (database), **Secrets Extension** (OpenBao for runtime secret injection), **Traces Observer** (trace querying), and **Evaluation Engine** (automated agent evaluations).

This guide installs both layers on a single-node k3d cluster on your local machine.

info

This setup is for **development and exploration**. For production deployments, see the [Production Considerations](#production-considerations) section.

## Prerequisites[​](#prerequisites "Direct link to Prerequisites")

### Hardware[​](#hardware "Direct link to Hardware")

| Resource | Minimum      |
| -------- | ------------ |
| RAM      | 8 GB         |
| CPU      | 4 cores      |
| Disk     | \~10 GB free |

### Required Tools[​](#required-tools "Direct link to Required Tools")

| Tool                                               | Version | Purpose                    |
| -------------------------------------------------- | ------- | -------------------------- |
| [Docker](https://docs.docker.com/get-docker/)      | v26.0+  | Container runtime          |
| [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+  | Kubernetes CLI             |
| [Helm](https://helm.sh/docs/intro/install/)        | v3.12+  | Kubernetes package manager |
| [k3d](https://k3d.io/#installation)                | v5.8+   | Local Kubernetes clusters  |

Verify all tools:

```
docker --version && kubectl version --client && helm version && k3d version
```

macOS with Colima

If you use Colima instead of Docker Desktop, start it with sufficient resources:

```
colima start --vm-type=vz --vz-rosetta --cpu 4 --memory 8
```

Prefix the cluster creation command in Step 1 with `K3D_FIX_DNS=0`.

### Required Ports[​](#required-ports "Direct link to Required Ports")

The following host ports must be free before installation:

**Agent Manager**

| Port | Purpose            |
| ---- | ------------------ |
| 3000 | Console UI         |
| 9000 | API                |
| 9243 | Internal API       |
| 9098 | Traces Observer    |
| 8084 | AI Gateway (HTTP)  |
| 8243 | AI Gateway (HTTPS) |

**OpenChoreo**

| Port  | Purpose                             |
| ----- | ----------------------------------- |
| 6550  | Kubernetes API                      |
| 8080  | Control Plane (HTTP)                |
| 8443  | Control Plane (HTTPS)               |
| 19080 | Data Plane Gateway (HTTP)           |
| 19443 | Data Plane Gateway (HTTPS)          |
| 10082 | Container Registry (Workflow Plane) |
| 11080 | Observer API                        |
| 11082 | OpenSearch API                      |
| 11085 | OpenSearch HTTPS                    |
| 21893 | OTel Collector                      |
| 22893 | Observability Gateway (HTTP)        |
| 22894 | Observability Gateway (HTTPS)       |

***

## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform")

OpenChoreo organises its infrastructure into four planes, each handling a different concern:

* **Control Plane** — API server and configuration management for the platform
* **Data Plane** — runs deployed workloads and API gateways
* **Workflow Plane** — builds and CI pipelines for agent deployments
* **Observability Plane** — trace, log, and metrics collection via OpenSearch

This phase installs all four with Agent Manager-specific configuration. **Estimated time: \~15-20 minutes.**

### Step 1: Create k3d Cluster[​](#step-1-create-k3d-cluster "Direct link to Step 1: Create k3d Cluster")

Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings:

```
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/quick-start/k3d-config.yaml \
  | k3d cluster create --config=-
```

Set the kubectl context:

```
k3d kubeconfig merge amp-local --kubeconfig-merge-default
kubectl config use-context k3d-amp-local
```

Generate machine IDs for Fluent Bit log collection:

```
for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do
  docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id"
done
```

Verify

```
kubectl cluster-info --context k3d-amp-local
# Should show cluster running at https://0.0.0.0:6550
```

### Step 2: Apply CoreDNS Configuration[​](#step-2-apply-coredns-configuration "Direct link to Step 2: Apply CoreDNS Configuration")

Enables `*.openchoreo.localhost` DNS resolution inside the cluster:

```
kubectl apply -f https://raw.githubusercontent.com/openchoreo/openchoreo/v1.0.0-rc.1/install/k3d/common/coredns-custom.yaml
```

### Step 3: Install Cluster Prerequisites[​](#step-3-install-cluster-prerequisites "Direct link to Step 3: Install Cluster Prerequisites")

These are infrastructure components that OpenChoreo depends on. You only install them once per cluster.

**Gateway API CRDs** — standard Kubernetes resources for managing network gateways and routing:

```
kubectl apply --server-side \
  -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml
```

**cert-manager (v1.19.2)** — automates TLS certificate issuance and renewal:

```
helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true \
  --wait --timeout 180s
```

**External Secrets Operator (v1.3.2)** — syncs secrets from external stores (like OpenBao) into Kubernetes:

```
helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 1.3.2 \
  --set installCRDs=true \
  --wait --timeout 180s
```

**kgateway (v2.2.1)** — the network gateway for OpenChoreo planes:

```
helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
  --create-namespace \
  --namespace openchoreo-control-plane \
  --version v2.2.1

helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --version v2.2.1 \
  --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true
```

### Step 4: Setup Secrets Store (OpenBao)[​](#step-4-setup-secrets-store-openbao "Direct link to Step 4: Setup Secrets Store (OpenBao)")

OpenBao provides secret management for the Workflow Plane and deployed agents:

```
helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \
  --namespace openbao \
  --create-namespace \
  --version 0.25.6 \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-openbao.yaml \
  --timeout 180s

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s
```

Configure the External Secrets ClusterSecretStore:

```
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets-openbao
  namespace: openbao
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: default
spec:
  provider:
    vault:
      server: "http://openbao.openbao.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "openchoreo-secret-writer-role"
          serviceAccountRef:
            name: "external-secrets-openbao"
            namespace: "openbao"
EOF
```

### Step 5: Install OpenChoreo Control Plane[​](#step-5-install-openchoreo-control-plane "Direct link to Step 5: Install OpenChoreo Control Plane")

The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension:

```
helm install openchoreo-control-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \
  --version 1.0.0-rc.1 \
  --namespace openchoreo-control-plane \
  --create-namespace \
  --timeout 600s \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-cp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-control-plane --timeout=600s
```

What the values file configures

* Backstage disabled (AMP provides its own console)
* OIDC pointing to AMP Thunder Extension (`thunder.amp.localhost:8080`)
* OpenChoreo API hostname: `api.openchoreo.localhost`
* Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled

### Step 6: Setup Data Plane[​](#step-6-setup-data-plane "Direct link to Step 6: Setup Data Plane")

Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace:

```
kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f -
```

```
helm install openchoreo-data-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \
  --version 1.0.0-rc.1 \
  --namespace openchoreo-data-plane \
  --create-namespace \
  --timeout 600s \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-dp.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-data-plane --timeout=600s
```

Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is:

```
CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterDataPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$CA_CERT" | sed 's/^/        /')
  gateway:
    ingress:
      external:
        name: gateway-default
        namespace: openchoreo-data-plane
        http:
          host: "openchoreoapis.localhost"
          listenerName: http
          port: 19080
        https:
          host: "openchoreoapis.localhost"
          listenerName: https
          port: 19443
  secretStoreRef:
    name: amp-openbao-store
EOF
```

### Step 7: Setup Workflow Plane[​](#step-7-setup-workflow-plane "Direct link to Step 7: Setup Workflow Plane")

Copy the CA certificate:

```
kubectl create namespace openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-workflow-plane --dry-run=client -o yaml | kubectl apply -f -
```

Install the container registry and the Workflow Plane:

```
helm upgrade --install registry docker-registry \
  --repo https://twuni.github.io/docker-registry.helm \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --values https://raw.githubusercontent.com/openchoreo/openchoreo/v1.0.0-rc.1/install/k3d/single-cluster/values-registry.yaml \
  --timeout 120s

helm install openchoreo-workflow-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-workflow-plane \
  --version 1.0.0-rc.1 \
  --namespace openchoreo-workflow-plane \
  --create-namespace \
  --timeout 600s
```

Register the Workflow Plane:

```
BP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-workflow-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ClusterWorkflowPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  secretStoreRef:
    name: default
  clusterAgent:
    clientCA:
      value: |
$(echo "$BP_CA_CERT" | sed 's/^/        /')
EOF

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-workflow-plane --timeout=600s
```

### Step 8: Setup Observability Plane[​](#step-8-setup-observability-plane "Direct link to Step 8: Setup Observability Plane")

Copy the CA certificate:

```
kubectl create namespace openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -

CA_CRT=$(kubectl get secret cluster-gateway-ca \
  -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)
kubectl create configmap cluster-gateway-ca \
  --from-literal=ca.crt="$CA_CRT" \
  -n openchoreo-observability-plane --dry-run=client -o yaml | kubectl apply -f -
```

Create the ExternalSecrets for OpenSearch and Observer credentials:

```
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF
```

Wait for the ExternalSecrets to sync:

```
kubectl wait -n openchoreo-observability-plane \
  --for=condition=Ready externalsecret/opensearch-admin-credentials \
  externalsecret/observer-secret --timeout=60s
```

Apply the custom OpenTelemetry Collector ConfigMap (required for trace ingestion):

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/oc-collector-configmap.yaml \
  -n openchoreo-observability-plane
```

Install the Observability Plane:

```
helm install openchoreo-observability-plane \
  oci://ghcr.io/openchoreo/helm-charts/openchoreo-observability-plane \
  --version 1.0.0-rc.1 \
  --namespace openchoreo-observability-plane \
  --create-namespace \
  --timeout 25m \
  --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-op.yaml

kubectl wait --for=condition=Available \
  deployment --all -n openchoreo-observability-plane --timeout=300s
```

Install observability modules (logs, metrics, tracing):

```
# Logs module
helm upgrade --install observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.8 \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --timeout 10m

# Enable Fluent Bit log collection
helm upgrade observability-logs-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \
  --namespace openchoreo-observability-plane \
  --version 0.3.8 \
  --reuse-values \
  --set fluent-bit.enabled=true \
  --timeout 10m

# Metrics module
helm upgrade --install observability-metrics-prometheus \
  oci://ghcr.io/openchoreo/helm-charts/observability-metrics-prometheus \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.2.4 \
  --timeout 10m

# Tracing module (uses the custom OTel Collector ConfigMap)
helm upgrade --install observability-traces-opensearch \
  oci://ghcr.io/openchoreo/helm-charts/observability-tracing-opensearch \
  --create-namespace \
  --namespace openchoreo-observability-plane \
  --version 0.3.7 \
  --set openSearch.enabled=false \
  --set openSearchSetup.openSearchSecretName="opensearch-admin-credentials" \
  --set opentelemetry-collector.configMap.existingName="amp-opentelemetry-collector-config" \
  --timeout 10m
```

Register the Observability Plane and link it to other planes:

```
OP_CA_CERT=$(kubectl get secret cluster-agent-tls \
  -n openchoreo-observability-plane -o jsonpath='{.data.ca\.crt}' | base64 -d)

kubectl apply -f - <<EOF
apiVersion: openchoreo.dev/v1alpha1
kind: ObservabilityPlane
metadata:
  name: default
  namespace: default
spec:
  planeID: "default"
  clusterAgent:
    clientCA:
      value: |
$(echo "$OP_CA_CERT" | sed 's/^/        /')
  observerURL: http://observer.openchoreo.localhost:11080
EOF

# Link Data Plane to Observability Plane
kubectl patch clusterdataplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'

# Link Workflow Plane to Observability Plane
kubectl patch clusterworkflowplane default -n default --type merge \
  -p '{"spec":{"observabilityPlaneRef":{"kind":"ClusterObservabilityPlane","name":"default"}}}'
```

### Step 9: Verify OpenChoreo Installation[​](#step-9-verify-openchoreo-installation "Direct link to Step 9: Verify OpenChoreo Installation")

Before proceeding to Phase 2, confirm all planes are running:

```
echo "--- Control Plane ---"
kubectl get pods -n openchoreo-control-plane
echo "--- Data Plane ---"
kubectl get pods -n openchoreo-data-plane
echo "--- Workflow Plane ---"
kubectl get pods -n openchoreo-workflow-plane
echo "--- Observability Plane ---"
kubectl get pods -n openchoreo-observability-plane
echo "--- Plane Registrations ---"
kubectl get clusterdataplane,clusterworkflowplane,observabilityplane -n default
```

All pods should be in `Running` or `Completed` state.

***

## Phase 2: Agent Manager Installation[​](#phase-2-agent-manager-installation "Direct link to Phase 2: Agent Manager Installation")

With OpenChoreo running, you can now install the Agent Manager components — the API, console, identity provider, and extensions that provide the AI agent management capabilities.

<!-- -->

The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order:

1. **Agent Manager Core :** Gateway Operator, Thunder Extension, Agent Manager and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it.
2. **Extensions :** Secret Management, Observability, Evaluation extensions and the AI Gateway Extension.

#### Configuration Variables[​](#configuration-variables "Direct link to Configuration Variables")

Set these once before running the install commands:

```
export VERSION="0.11.0"
export HELM_CHART_REGISTRY="ghcr.io/wso2"
export AMP_NS="wso2-amp"
export BUILD_CI_NS="openchoreo-workflow-plane"
export OBSERVABILITY_NS="openchoreo-observability-plane"
export DEFAULT_NS="default"
export DATA_PLANE_NS="openchoreo-data-plane"
export SECRETS_NS="amp-secrets"
export THUNDER_NS="amp-thunder"

# Observability endpoint for the console.
# k3d default below works out of the box. For managed clusters,
# replace with your observability gateway LoadBalancer address:
#   export INSTRUMENTATION_URL="http://<obs-gateway-lb-ip>:22893/otel"
export INSTRUMENTATION_URL="http://localhost:22893/otel"
```

***

### Core Components[​](#core-components "Direct link to Core Components")

Install these in order — each depends on the one before it.

#### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator")

Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane.

```
helm install gateway-operator \
  oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \
  --version 0.4.0 \
  --namespace ${DATA_PLANE_NS} \
  --set logging.level=debug \
  --set gateway.helm.chartVersion=0.9.0 \
  --timeout 600s
```

Wait for the operator to be ready:

```
kubectl wait --for=condition=Available \
  deployment -l app.kubernetes.io/name=gateway-operator \
  -n ${DATA_PLANE_NS} --timeout=300s
```

Apply the Gateway Operator configuration (JWT/JWKS authentication and rate limiting):

<!-- -->

Download the config, rewrite the JWKS URI for k3d (so the gateway can reach the AMP API via the Docker host network), then apply: \`\`\`bash curl -fsSL https\://raw\.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/api-platform-operator-full-config.yaml \ | sed 's|http\://amp-api.wso2-amp.svc.cluster.local:9000/auth/external/jwks.json|http\://host.k3d.internal:9000/auth/external/jwks.json|g' \ | kubectl apply -f - \`\`\`

<!-- -->

Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent:

```
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: wso2-api-platform-gateway-module
rules:
  - apiGroups: ["gateway.api-platform.wso2.com"]
    resources: ["restapis", "apigateways"]
    verbs: ["*"]
  - apiGroups: ["gateway.kgateway.dev"]
    resources: ["backends"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: wso2-api-platform-gateway-module
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: wso2-api-platform-gateway-module
subjects:
  - kind: ServiceAccount
    name: cluster-agent-dataplane
    namespace: ${DATA_PLANE_NS}
EOF
```

Deploy the observability gateway and trace API:

```
kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/obs-gateway.yaml

kubectl wait --for=condition=Programmed \
  apigateway/obs-gateway -n ${DATA_PLANE_NS} --timeout=180s

kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/otel-collector-rest-api.yaml

kubectl wait --for=condition=Programmed \
  restapi/traces-api-secure -n ${DATA_PLANE_NS} --timeout=120s
```

Verify

```
kubectl get apigateway obs-gateway -n ${DATA_PLANE_NS}
# STATUS should show "Programmed"
```

#### Step 2: Thunder Extension (Identity Provider)[​](#step-2-thunder-extension-identity-provider "Direct link to Step 2: Thunder Extension (Identity Provider)")

Provides authentication and user management for the Agent Manager platform — login, API keys, and OAuth token exchange.

```
helm install amp-thunder-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \
  --version ${VERSION} \
  --namespace ${THUNDER_NS} \
  --create-namespace \
  --timeout 1800s
```

Verify

```
kubectl get pods -n ${THUNDER_NS}
# All pods should be Running
```

#### Step 3: Agent Manager (API + Console + PostgreSQL)[​](#step-3-agent-manager-api--console--postgresql "Direct link to Step 3: Agent Manager (API + Console + PostgreSQL)")

The core platform: a Go API server, a React web console, and a PostgreSQL database.

```
helm install amp \
  oci://${HELM_CHART_REGISTRY}/wso2-agent-manager \
  --version ${VERSION} \
  --namespace ${AMP_NS} \
  --create-namespace \
  --set console.config.instrumentationUrl="${INSTRUMENTATION_URL}" \
  --timeout 1800s
```

info

`INSTRUMENTATION_URL` was set in the [Configuration Variables](#configuration-variables) above. For managed clusters, update it to your observability gateway LoadBalancer address before running this command.

Wait for all components:

```
# PostgreSQL
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/amp-postgresql -n ${AMP_NS} --timeout=600s

# API server
kubectl wait --for=condition=Available \
  deployment/amp-api -n ${AMP_NS} --timeout=600s

# Console
kubectl wait --for=condition=Available \
  deployment/amp-console -n ${AMP_NS} --timeout=600s
```

Verify

```
kubectl get pods -n ${AMP_NS}
# Expected: amp-postgresql-0 (Running), amp-api-xxx (Running), amp-console-xxx (Running)
```

#### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources")

Creates the default Organization, Project, Environment, and DeploymentPipeline resources that the console needs on first login.

```
helm install amp-platform-resources \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \
  --version ${VERSION} \
  --namespace ${DEFAULT_NS} \
  --timeout 1800s
```

***

### Extensions[​](#extensions "Direct link to Extensions")

These can be installed in any order after Core is ready.

#### Step 5: Secrets Extension (OpenBao)[​](#step-5-secrets-extension-openbao "Direct link to Step 5: Secrets Extension (OpenBao)")

Provides runtime secret injection for deployed agents. Uses [OpenBao](https://openbao.org/) as the secrets backend.

```
helm install amp-secrets \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-secrets-extension \
  --version ${VERSION} \
  --namespace ${SECRETS_NS} \
  --create-namespace \
  --set openbao.server.dev.enabled=true \
  --timeout 600s
```

```
kubectl wait --for=jsonpath='{.status.readyReplicas}'=1 \
  statefulset/openbao -n ${SECRETS_NS} --timeout=300s
```

warning

Dev mode uses an in-memory backend — secrets are lost on restart. For production, disable dev mode and configure persistent storage.

#### Step 6: Observability Extension (Traces Observer)[​](#step-6-observability-extension-traces-observer "Direct link to Step 6: Observability Extension (Traces Observer)")

Deploys the Traces Observer service that queries and serves trace data to the console.

Create the required ExternalSecrets first:

```
kubectl apply -f - <<'EOF'
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: opensearch-admin-credentials
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: opensearch-admin-credentials
  data:
  - secretKey: username
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: password
    remoteRef:
      key: opensearch-password
      property: value
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: observer-secret
  namespace: openchoreo-observability-plane
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: default
  target:
    name: observer-secret
  data:
  - secretKey: OPENSEARCH_USERNAME
    remoteRef:
      key: opensearch-username
      property: value
  - secretKey: OPENSEARCH_PASSWORD
    remoteRef:
      key: opensearch-password
      property: value
  - secretKey: UID_RESOLVER_OAUTH_CLIENT_SECRET
    remoteRef:
      key: observer-oauth-client-secret
      property: value
EOF
```

Install the extension:

```
helm install amp-observability-traces \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \
  --version ${VERSION} \
  --namespace ${OBSERVABILITY_NS} \
  --timeout 1800s

kubectl wait --for=condition=Available \
  deployment/amp-traces-observer -n ${OBSERVABILITY_NS} --timeout=600s
```

#### Step 7: Evaluation Extension[​](#step-7-evaluation-extension "Direct link to Step 7: Evaluation Extension")

Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces.

```
helm install amp-evaluation-extension \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \
  --version ${VERSION} \
  --namespace ${BUILD_CI_NS} \
  --timeout 1800s
```

info

The default `publisher.apiKey` must match `publisherApiKey.value` in the Agent Manager chart. Both default to `amp-internal-api-key`.

#### Step 8: AI Gateway Extension[​](#step-8-ai-gateway-extension "Direct link to Step 8: AI Gateway Extension")

Registers the AI Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange.

```
helm install amp-ai-gateway \
  oci://${HELM_CHART_REGISTRY}/wso2-amp-ai-gateway-extension \
  --version ${VERSION} \
  --namespace ${DATA_PLANE_NS} \
  --set apiGateway.controlPlane.host="amp-api-gateway-manager.${AMP_NS}.svc.cluster.local:9243" \
  --set agentManager.apiUrl="http://amp-api.${AMP_NS}.svc.cluster.local:9000/api/v1" \
  --set agentManager.idp.tokenUrl="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090/oauth2/token" \
  --timeout 1800s

kubectl wait --for=condition=complete job/amp-gateway-bootstrap \
  -n ${DATA_PLANE_NS} --timeout=300s
```

Verify

```
kubectl get jobs -n ${DATA_PLANE_NS} | grep amp-gateway-bootstrap
# STATUS should show "Complete"
```

***

## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform")

Run a full status check to confirm everything is running:

```
# All pods across key namespaces
kubectl get pods -n openchoreo-control-plane
kubectl get pods -n openchoreo-data-plane
kubectl get pods -n openchoreo-workflow-plane
kubectl get pods -n openchoreo-observability-plane
kubectl get pods -n wso2-amp
kubectl get pods -n amp-thunder
kubectl get pods -n amp-secrets

# Helm releases
helm list -A | grep -E 'openchoreo|amp|gateway'
```

| Service                           | URL                            |
| --------------------------------- | ------------------------------ |
| **Agent Manager Console**         | <http://localhost:3000>        |
| **Agent Manager API**             | <http://localhost:9000>        |
| **Observability Gateway (HTTP)**  | <http://localhost:22893/otel>  |
| **Observability Gateway (HTTPS)** | <https://localhost:22894/otel> |

**Default credentials:** `admin` / `admin`

### Self-Signed Certificate for OTEL HTTPS[​](#self-signed-certificate-for-otel-https "Direct link to Self-Signed Certificate for OTEL HTTPS")

If your OTEL exporters encounter certificate errors on the HTTPS endpoint:

```
kubectl get secret obs-gateway-gateway-controller-tls \
  -n openchoreo-data-plane \
  -o jsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt

export OTEL_EXPORTER_OTLP_CERTIFICATE=$(pwd)/ca.crt
```

***

## Cleanup[​](#cleanup "Direct link to Cleanup")

Delete the entire k3d cluster and all resources:

```
k3d cluster delete amp-local
```

***

## Production Considerations[​](#production-considerations "Direct link to Production Considerations")

This installation is designed for development and exploration. For production:

1. **Identity provider** — Replace Thunder dev mode with a proper IdP (Asgardeo, Auth0, Okta)
2. **TLS** — Replace self-signed certificates with CA-signed certificates
3. **Secrets backend** — Disable OpenBao dev mode; configure persistent storage and proper auth
4. **Observability storage** — Configure persistent volumes for OpenSearch
5. **Resource sizing** — Adjust requests/limits based on workload
6. **High availability** — Deploy multiple replicas of critical components
7. **Security hardening** — Apply network policies, RBAC, pod security standards

***

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Pods stuck in Pending

Usually a resource constraint. Check node capacity:

```
kubectl describe pod <pod-name> -n <namespace>
kubectl top nodes
```

Increase Colima/Docker Desktop resources if needed.

Gateway not becoming Programmed

```
kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator
kubectl describe apigateway obs-gateway -n openchoreo-data-plane
```

Plane registration issues

```
kubectl get clusterdataplane default -n default -o yaml
kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane
```

OpenSearch connectivity issues

```
kubectl get pods -n openchoreo-observability-plane -l app=opensearch
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200
```

Port already in use

Find the process occupying the port and stop it:

```
lsof -i :<port>
```

***

## Reference: Configuration Files[​](#reference-configuration-files "Direct link to Reference: Configuration Files")

All Agent Manager-specific configuration files used in this guide:

| File                                                                                                                                                                 | Purpose                                                       |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| [k3d-config.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/quick-start/k3d-config.yaml)                                          | k3d cluster with all required port mappings                   |
| [values-cp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-cp.yaml)                                         | Control Plane — Backstage disabled, AMP Thunder OIDC          |
| [values-dp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-dp.yaml)                                         | Data Plane — gateway ports, Fluent Bit config                 |
| [values-op.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-op.yaml)                                         | Observability Plane — standalone OpenSearch, AMP Thunder OIDC |
| [values-openbao.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/single-cluster/values-openbao.yaml)                               | OpenBao — dev mode, Kubernetes auth, pre-seeded secrets       |
| [oc-collector-configmap.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/oc-collector-configmap.yaml)                       | Custom OTel Collector ConfigMap for trace ingestion           |
| [api-platform-operator-full-config.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/api-platform-operator-full-config.yaml) | Gateway Operator — JWT auth, rate limiting                    |
| [obs-gateway.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/obs-gateway.yaml)                                             | Observability API Gateway resource                            |
| [otel-collector-rest-api.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.11.0/deployments/values/otel-collector-rest-api.yaml)                     | OTel Collector REST API resource                              |