# On k3d (Locally) Install the Agent Manager on a local Kubernetes cluster using [k3d](https://k3d.io/) — a lightweight tool that runs Kubernetes inside Docker. Just want it running? The [Quick Start Guide](/agent-manager/docs/next/get-started/quick-start/.md) installs everything in a single command using a dev container. Use this page when you want to understand the full setup or need to customize it. Stronger agent isolation tiers Agents run sandboxed under the standard **runc** runtime by default. Agent Manager also supports stronger per-environment isolation tiers — **gVisor** (userspace kernel) and **Kata Containers** (per-agent VM) — but they have hardware/OS requirements and need a dedicated node. For more information, see the [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) setup guides. ## What You Will Get[​](#what-you-will-get "Direct link to What You Will Get") Agent Manager is a two-layer system installed in two phases: * **Phase 1 — OpenChoreo (base layer):** [OpenChoreo](https://openchoreo.dev) is an open-source platform that provides the Kubernetes infrastructure Agent Manager runs on. It consists of four planes: a **Control Plane** for API and configuration, a **Data Plane** for running workloads and gateways, a **Workflow Plane** for builds and CI pipelines, and an **Observability Plane** for traces, logs, and metrics via OpenSearch. * **Phase 2 — Agent Manager :** The AI agent management platform installed on top of OpenChoreo. It includes **Thunder** (identity provider — installed with OpenChoreo since it provides JWT validation for all planes), the **Console** (web UI), **AMP API** (backend), **AI Gateway**, **PostgreSQL** (database), **Secrets Extension** (OpenBao for runtime secret injection), **Agent Manager Observer** (traces, logs, and metrics), and **Evaluation Engine** (automated agent evaluations). This guide installs both layers on a single-node k3d cluster on your local machine. info This setup is for **development and exploration**. For production deployments, see the [Production Considerations](#production-considerations) section. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") ### Hardware[​](#hardware "Direct link to Hardware") | Resource | Minimum | | -------- | ------------ | | RAM | 8 GB | | CPU | 4 cores | | Disk | \~10 GB free | ### Required Tools[​](#required-tools "Direct link to Required Tools") | Tool | Version | Purpose | | -------------------------------------------------- | ------- | -------------------------- | | [Docker](https://docs.docker.com/get-docker/) | v26.0+ | Container runtime | | [kubectl](https://kubernetes.io/docs/tasks/tools/) | v1.32+ | Kubernetes CLI | | [Helm](https://helm.sh/docs/intro/install/) | v3.12+ | Kubernetes package manager | | [k3d](https://k3d.io/#installation) | v5.8+ | Local Kubernetes clusters | Verify all tools: ``` docker --version && kubectl version --client && helm version && k3d version ``` macOS with Colima If you use Colima instead of Docker Desktop, start a dedicated `agent-manager` profile so the install does not interfere with your default Colima instance: ``` colima start --profile agent-manager \ --vm-type=vz --vz-rosetta --network-address \ --cpu 4 --memory 8 ``` `--network-address` assigns the VM a stable IP so the cluster keeps working after network changes. Prefix the cluster creation command in Step 1 with `K3D_FIX_DNS=0`. ### Required Ports[​](#required-ports "Direct link to Required Ports") The following host ports must be free before installation. The Agent Manager services do not need host ports of their own: the Console and API are served through the Control Plane gateway (8080), the Agent Manager Observer through the Observability Plane gateway (11080), and trace ingestion through the Data Plane gateway (19080). | Port | Purpose | | ----- | ------------------------------------------------------------------------------------------------------- | | 6550 | Kubernetes API | | 8080 | Control Plane Gateway (HTTP) — Console, API, Thunder, OpenChoreo API, external AI gateway control plane | | 8443 | Control Plane Gateway (HTTPS) | | 19080 | Data Plane Gateway (HTTP) — deployed agents, OTLP `/otel` ingest | | 19443 | Data Plane Gateway (HTTPS) | | 10082 | Container Registry (Workflow Plane) | | 11080 | Observability Plane Gateway (HTTP) — Agent Manager Observer, Observer API | | 11082 | OpenSearch API | | 11085 | OpenSearch HTTPS | *** ## Phase 1: OpenChoreo Platform[​](#phase-1-openchoreo-platform "Direct link to Phase 1: OpenChoreo Platform") OpenChoreo organises its infrastructure into four planes, each handling a different concern: * **Control Plane** — API server and configuration management for the platform * **Data Plane** — runs deployed workloads and API gateways * **Workflow Plane** — builds and CI pipelines for agent deployments * **Observability Plane** — trace, log, and metrics collection via OpenSearch This phase installs all four with Agent Manager-specific configuration. **Estimated time: \~15-20 minutes.** ### Step 1: Create k3d Cluster[​](#step-1-create-k3d-cluster "Direct link to Step 1: Create k3d Cluster") Create the cluster using the Agent Manager cluster configuration, which includes all required port mappings: ``` curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/quick-start/k3d-config.yaml \ | k3d cluster create --config=- ``` Set the kubectl context: ``` k3d kubeconfig merge amp-local --kubeconfig-merge-default kubectl config use-context k3d-amp-local ``` Generate machine IDs for Fluent Bit log collection: ``` for NODE in $(k3d node list -o json | grep -o '"name":"[^"]*"' | sed 's/"name":"//;s/"//' | grep "^k3d-amp-local-"); do docker exec ${NODE} sh -c "cat /proc/sys/kernel/random/uuid | tr -d '-' > /etc/machine-id" done ``` Verify ``` kubectl cluster-info --context k3d-amp-local # Should show cluster running at https://0.0.0.0:6550 ``` ### Step 2: Apply CoreDNS Configuration[​](#step-2-apply-coredns-configuration "Direct link to Step 2: Apply CoreDNS Configuration") Enables `*.openchoreo.localhost`, `*.amp.localhost`, `*.agentmanager.localhost`, `*.am-gateway.localhost`, and `*.gateway.localhost` DNS resolution inside the cluster — required for in-cluster lookups of the Console/API/Thunder hostnames as well as agent-to-agent and AI gateway traffic: ``` kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/k8s/coredns-amp-custom.yaml ``` CoreDNS's reload plugin can miss the override if the ConfigMap mounts after the pod's initial parse, so restart it to make sure the rewrite rules are active before anything depends on them: ``` kubectl rollout restart deployment/coredns -n kube-system kubectl rollout status deployment/coredns -n kube-system --timeout=60s ``` ### Step 3: Install Cluster Prerequisites[​](#step-3-install-cluster-prerequisites "Direct link to Step 3: Install Cluster Prerequisites") These are infrastructure components that OpenChoreo depends on. You only install them once per cluster. **Gateway API CRDs** — standard Kubernetes resources for managing network gateways and routing: ``` kubectl apply --server-side \ -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml ``` **cert-manager (v1.19.2)** — automates TLS certificate issuance and renewal: ``` helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \ --namespace cert-manager \ --create-namespace \ --version v1.19.2 \ --set crds.enabled=true \ --wait --timeout 180s ``` **External Secrets Operator (v1.3.2)** — syncs secrets from external stores (like OpenBao) into Kubernetes: ``` helm upgrade --install external-secrets oci://ghcr.io/external-secrets/charts/external-secrets \ --namespace external-secrets \ --create-namespace \ --version 1.3.2 \ --set installCRDs=true \ --wait --timeout 180s ``` **kgateway (v2.2.1)** — the network gateway for OpenChoreo planes: ``` helm upgrade --install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \ --create-namespace \ --namespace openchoreo-control-plane \ --version v2.2.1 helm upgrade --install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \ --namespace openchoreo-control-plane \ --create-namespace \ --version v2.2.1 \ --set controller.extraEnv.KGW_ENABLE_GATEWAY_API_EXPERIMENTAL_FEATURES=true ``` ### Step 4: Setup Secrets Store (OpenBao)[​](#step-4-setup-secrets-store-openbao "Direct link to Step 4: Setup Secrets Store (OpenBao)") OpenBao provides secret management for the Workflow Plane and deployed agents: ``` helm upgrade --install openbao oci://ghcr.io/openbao/charts/openbao \ --namespace openbao \ --create-namespace \ --version 0.25.6 \ --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-openbao.yaml \ --timeout 180s kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=openbao -n openbao --timeout=120s ``` Configure the External Secrets ClusterSecretStore: ``` kubectl apply -f - <<'EOF' apiVersion: v1 kind: ServiceAccount metadata: name: external-secrets-openbao namespace: openbao --- apiVersion: external-secrets.io/v1 kind: ClusterSecretStore metadata: name: default spec: provider: vault: server: "http://openbao.openbao.svc:8200" path: "secret" version: "v2" auth: kubernetes: mountPath: "kubernetes" role: "openchoreo-secret-writer-role" serviceAccountRef: name: "external-secrets-openbao" namespace: "openbao" EOF ``` ### Step 5: Configuration Variables[​](#step-5-configuration-variables "Direct link to Step 5: Configuration Variables") Set these variables before continuing. They are used by the remaining Phase 1 steps and all of Phase 2. ``` export VERSION="0.0.0-dev" export HELM_CHART_REGISTRY="ghcr.io/wso2" export AMP_NS="wso2-amp" export BUILD_CI_NS="openchoreo-workflow-plane" export OBSERVABILITY_NS="openchoreo-observability-plane" export DEFAULT_NS="default" export DATA_PLANE_NS="openchoreo-data-plane" export THUNDER_NS="amp-thunder" export OPENCHOREO_INTERNAL_URL="http://openchoreo-api.openchoreo-control-plane.svc.cluster.local:8080" export THUNDER_PUBLIC_URL="http://thunder.amp.localhost:8080" export THUNDER_INTERNAL_URL="http://amp-thunder-extension-service.${THUNDER_NS}.svc.cluster.local:8090" export CONSOLE_PUBLIC_URL="http://console.amp.localhost:8080" export API_PUBLIC_URL="http://api.amp.localhost:8080" export OBS_API_PUBLIC_URL="http://traces.amp.localhost:11080" # Bare hostnames (no scheme/port) — used as the HTTPRoute hostnames on the plane gateways export CONSOLE_PUBLIC_HOST="console.amp.localhost" export API_PUBLIC_HOST="api.amp.localhost" export OBS_API_PUBLIC_HOST="traces.amp.localhost" export INSTRUMENTATION_URL="http://default-default.gateway.localhost:19080/otel" ``` info In k3d, every management endpoint is accessed through a plane gateway using a hostname: Thunder, the Console, and the API via the Control Plane kgateway on port 8080 (`thunder.amp.localhost`, `console.amp.localhost`, `api.amp.localhost`), and the Agent Manager Observer via the Observability Plane kgateway on port 11080 (`traces.amp.localhost`). These match the chart defaults and the Control Plane / Observability Plane OIDC configuration. No port-forwarding is needed; `*.localhost` names resolve to the loopback address in browsers and curl. ### Step 6: Install Thunder Extension (Identity Provider)[​](#step-6-install-thunder-extension-identity-provider "Direct link to Step 6: Install Thunder Extension (Identity Provider)") Thunder provides authentication and user management for the entire platform — login, API keys, and OAuth token exchange. It must be installed before the Control Plane because the Control Plane, Observability Plane, and Agent Manager all validate JWTs issued by Thunder. ``` helm install amp-thunder-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-thunder-extension \ --version ${VERSION} \ --namespace ${THUNDER_NS} \ --create-namespace \ --timeout 1800s kubectl wait --for=condition=Available \ deployment -l app.kubernetes.io/instance=amp-thunder-extension \ -n ${THUNDER_NS} --timeout=300s ``` info No MCP overrides are needed here: `thunder.bootstrap.agentManagerMcpBaseUrl` and `observerMcpBaseUrl` become the OAuth resource-server identifiers Thunder registers for the `/mcp` endpoints, and their chart defaults are already the k3d gateway URLs used throughout this guide. If you deviate from those hostnames, override both to match `agentManagerService.config.serverPublicURL` and `amObserver.publicUrl` — Thunder answers `invalid_target` unless an MCP client's requested `resource` matches a registered identifier exactly. Both are seeded by the pre-install bootstrap job only, so a `helm upgrade` will not re-seed them. warning Thunder persists its configuration (including the issuer URL) in a database on first boot. If you need to change `THUNDER_PUBLIC_URL` after installation, you must **uninstall the chart, delete its PVC, and reinstall** — a `helm upgrade` alone will not change the issuer in issued tokens. Verify ``` kubectl exec -n ${THUNDER_NS} deploy/amp-thunder-extension-deployment -- \ wget -qO- http://localhost:8090/.well-known/openid-configuration 2>/dev/null \ | grep -o '"issuer":"[^"]*"' # Expected: "issuer":"http://thunder.amp.localhost:8080" ``` ### Step 7: Install OpenChoreo Control Plane[​](#step-7-install-openchoreo-control-plane "Direct link to Step 7: Install OpenChoreo Control Plane") The Control Plane is configured with Backstage disabled (Agent Manager provides its own console) and OIDC pointing to the AMP Thunder Extension: ``` helm install openchoreo-control-plane \ oci://ghcr.io/openchoreo/helm-charts/openchoreo-control-plane \ --version 1.1.1 \ --namespace openchoreo-control-plane \ --create-namespace \ --timeout 600s \ --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-cp.yaml kubectl wait --for=condition=Available \ deployment --all -n openchoreo-control-plane --timeout=600s ``` Webhook race condition If the install fails with `no endpoints available for service "controller-manager-webhook-service"`, this is a known transient race in OpenChoreo v1.1.1. Wait for the control plane deployments to become ready, then rerun the `helm install` command. ``` kubectl wait --for=condition=Available deployment --all \ -n openchoreo-control-plane --timeout=300s ``` What the values file configures * Backstage disabled (AMP provides its own console) * OIDC pointing to AMP Thunder Extension (`thunder.amp.localhost:8080`) * OpenChoreo API hostname: `api.openchoreo.localhost` * Gateway on HTTP port 8080 / HTTPS port 8443, TLS disabled #### Patch the service-account entitlement claim (required with Thunder ≥ 0.45)[​](#patch-the-service-account-entitlement-claim-required-with-thunder--045 "Direct link to Patch the service-account entitlement claim (required with Thunder ≥ 0.45)") Thunder 0.45+ puts the client name in the `client_id` claim (its `sub` is an opaque UUID), but OpenChoreo 1.1.1 extracts service-account entitlements from `sub`, and the chart does not expose the setting. Without this patch, every service-to-service call is **silently unauthorized** — the API returns `200` with empty lists and `403`s while all pods look healthy, and the gateway bootstrap later fails with `Environment 'default' not found`. ``` patched_yaml=$(kubectl get configmap openchoreo-api-config -n openchoreo-control-plane -o yaml \ | sed -E "s/claim:[[:space:]]*['\"]?sub['\"]?/claim: client_id/g") echo "$patched_yaml" | kubectl apply --server-side --field-manager=helm --force-conflicts -f - kubectl rollout restart deployment/openchoreo-api -n openchoreo-control-plane kubectl rollout status deployment/openchoreo-api -n openchoreo-control-plane --timeout=120s for binding in $(kubectl get clusterauthzrolebindings.openchoreo.dev -o jsonpath='{.items[*].metadata.name}'); do claim=$(kubectl get clusterauthzrolebinding.openchoreo.dev "$binding" -o jsonpath='{.spec.entitlement.claim}') if [ "$claim" = "sub" ]; then kubectl patch clusterauthzrolebinding.openchoreo.dev "$binding" --type=merge \ -p '{"spec":{"entitlement":{"claim":"client_id"}}}' fi done ``` note Re-apply this patch after any `helm upgrade` of the Control Plane chart — the upgrade reverts both the ConfigMap and the bindings. The same patch is needed a second time for the Observability Plane's `observer-auth-config` in Step 10. ### Step 8: Setup Data Plane[​](#step-8-setup-data-plane "Direct link to Step 8: Setup Data Plane") Each plane needs the Control Plane's CA certificate to establish trusted communication. Copy it to the Data Plane namespace: ``` kubectl create namespace openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f - CA_CRT=$(kubectl get secret cluster-gateway-ca \ -n openchoreo-control-plane -o jsonpath='{.data.ca\.crt}' | base64 -d) kubectl create configmap cluster-gateway-ca \ --from-literal=ca.crt="$CA_CRT" \ -n openchoreo-data-plane --dry-run=client -o yaml | kubectl apply -f - ``` ``` helm install openchoreo-data-plane \ oci://ghcr.io/openchoreo/helm-charts/openchoreo-data-plane \ --version 1.1.1 \ --namespace openchoreo-data-plane \ --create-namespace \ --timeout 600s \ --values https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-dp.yaml kubectl wait --for=condition=Available \ deployment --all -n openchoreo-data-plane --timeout=600s ``` Register the Data Plane with the Control Plane. The command below automatically extracts the CA certificate and inserts it into the YAML — run the entire block as-is: ``` CA_CERT=$(kubectl get secret cluster-agent-tls \ -n openchoreo-data-plane -o jsonpath='{.data.ca\.crt}' | base64 -d) kubectl apply -f - < The Agent Manager installs as a set of Helm charts on top of OpenChoreo. The components fall into two groups based on install order: 1. **Agent Manager Core :** Gateway Operator, Agent Manager, Agent Sandbox Module and Platform Resources (agent component types, workflow templates etc). Each depends on the one before it. 2. **Extensions :** Secret Management, Observability, Evaluation extensions and the API Platform Gateway Extension. Prerequisites Thunder (identity provider) must be installed before proceeding — see the Thunder installation step in Phase 1. The variables `THUNDER_PUBLIC_URL`, `THUNDER_INTERNAL_URL`, `CONSOLE_PUBLIC_URL`, `API_PUBLIC_URL`, `OBS_API_PUBLIC_URL`, `CONSOLE_PUBLIC_HOST`, `API_PUBLIC_HOST`, `OBS_API_PUBLIC_HOST`, and `INSTRUMENTATION_URL` must be set from the Configuration Variables section. The production steps below additionally use `AGENTS_DOMAIN` — the domain deployed agents are served on, `agents.` — to build the gateway's hostname and virtual host. The `*_PUBLIC_HOST` variables carry the bare hostname (no scheme or port) and become the HTTPRoute hostnames on the plane gateways. *** ### Core Components[​](#core-components "Direct link to Core Components") Install these in order — each depends on the one before it. #### Step 1: Gateway Operator[​](#step-1-gateway-operator "Direct link to Step 1: Gateway Operator") Manages API Gateway resources and enables secure, authenticated trace ingestion into the Observability Plane. The gateway controller encrypts stored credentials at rest and will not start without a key, so create one before installing the operator: ``` kubectl create namespace ${DATA_PLANE_NS} --dry-run=client -o yaml | kubectl apply -f - openssl rand 32 > gateway-aesgcm.key kubectl create secret generic gateway-encryption-keys \ --namespace ${DATA_PLANE_NS} \ --from-file=default-aesgcm256-v1.bin=gateway-aesgcm.key rm -f gateway-aesgcm.key ``` The key is required, and the file name is fixed The controller looks for a specific path, `/app/data/aesgcm-keys/default-aesgcm256-v1.bin`, so the Secret key must be exactly `default-aesgcm256-v1.bin`. Without it the controller crash-loops on `failed to initialize key manager: encryption key file not found for version aesgcm256-v1`, and the gateway never programs — the rest of the platform stays healthy, so the only symptom is that agents cannot be invoked. This applies whatever `developmentMode` is set to. Earlier gateway releases auto-generated a key in development mode; from `1.2.0-beta` they do not. Store the key with the rest of your platform secrets. It encrypts credentials the gateway holds, and losing it means those entries cannot be decrypted. ``` helm install gateway-operator \ oci://ghcr.io/wso2/api-platform/helm-charts/gateway-operator \ --version 0.10.1 \ --namespace ${DATA_PLANE_NS} \ --set logging.level=debug \ --set gatewayApi.installStandardCRDs=false \ --set gateway.helm.chartVersion=1.2.0-beta \ --set gateway.values.gateway.controller.encryptionKeys.enabled=true \ --set gateway.values.gateway.controller.encryptionKeys.secretName=gateway-encryption-keys \ --timeout 600s ``` Pin the gateway chart, do not inherit it `gateway.helm.chartVersion` decides which gateway chart the operator deploys, and it is the only thing that decides it — the `APIGateway` resource carries no chart version. Operator `0.10.1` defaults to `1.2.0-alpha`, whose templates predate the controller reading its control-plane address from configuration while still shipping `1.2.0-beta` images. The result is a gateway that installs, programs, and serves traffic while never registering with Agent Manager, so it never appears in the gateway list. Pin `1.2.0-beta` as above so the chart and the images match. `gateway.values` is the operator's passthrough into that chart: anything set under it is merged into the values the gateway is deployed with, which is how the encryption keys above are wired. Wait for the operator to be ready: ``` kubectl wait --for=condition=Available \ deployment -l app.kubernetes.io/name=gateway-operator \ -n ${DATA_PLANE_NS} --timeout=300s ``` Grant RBAC for WSO2 API Platform CRDs to the Data Plane cluster-agent: ``` kubectl apply -f - < info `keyManager.issuer` must be the public Thunder URL — the chart default is the k3d hostname. The authorization server the API advertises in its RFC 9728 protected resource metadata is derived from it, so `agentManagerService.config.oauthAuthorizationServers` only needs setting if it must differ. Likewise the MCP audience entry is derived from `serverPublicURL` above, so `keyManager.audience` only needs setting to change the accepted client IDs. Verify ``` kubectl get pods -n ${AMP_NS} # Expected: amp-api-xxx (Running), amp-console-xxx (Running), plus # amp-postgresql-0 (Running) only when using the in-cluster database ``` #### Step 3: Agent Sandbox Module[​](#step-3-agent-sandbox-module "Direct link to Step 3: Agent Sandbox Module") Agents run as **sandboxed pods** managed by the [Agent Sandbox](https://agent-sandbox.sigs.k8s.io/) controller (`SandboxTemplate` / `SandboxWarmPool` resources) instead of plain Deployments. This module is **required** — without it, agent deployments cannot be rendered. ``` helm upgrade --install agent-sandbox \ oci://ghcr.io/openchoreo/helm-charts/agent-sandbox \ --version 0.1.1 \ --namespace ${DATA_PLANE_NS} \ --create-namespace \ --wait \ --timeout 10m \ --set namespace=openchoreo-control-plane \ --set dataPlaneNamespace=${DATA_PLANE_NS} \ --set dataPlaneServiceAccount=cluster-agent-dataplane \ --set upstream.version=v0.4.6 ``` Wait for the controller: ``` kubectl wait -n agent-sandbox-system \ --for=condition=available \ --timeout=180s \ deployment/agent-sandbox-controller ``` Verify ``` kubectl get crd \ sandboxtemplates.extensions.agents.x-k8s.io \ sandboxwarmpools.extensions.agents.x-k8s.io \ sandboxclaims.extensions.agents.x-k8s.io # All three CRDs should be listed kubectl get clusterrole openchoreo-agent-sandbox-access # RBAC granting the data-plane agent access to sandbox resources ``` Stronger isolation tiers With this module, agents run sandboxed under the standard **runc** runtime. Optionally, individual environments can run agents under **gVisor** or **Kata Containers** for stronger isolation — these have hardware/OS requirements and need a dedicated node. See the [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) isolation tier guides. #### Step 4: Platform Resources[​](#step-4-platform-resources "Direct link to Step 4: Platform Resources") Creates the default Organization, Project, Environment, DeploymentPipeline, and workflow template resources that the console needs on first login. This chart also configures the **container registry endpoint** used by build workflows to push agent images. ``` helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set apiPlatformGateway.namespace=${DATA_PLANE_NS} \ --timeout 1800s ``` Point the traits at the namespace the gateway is actually in Deployed agents reach the gateway runtime by DNS name, and both the trace-export endpoint and the route agent traffic is forwarded through are built from `apiPlatformGateway.namespace`. Leaving it empty derives the per-org-env convention `-`, which is what `add-environment.sh` creates — but [Step 7](#step-7-api-platform-gateway-extension) installs the gateway extension into `${DATA_PLANE_NS}`, matching that chart's own `apiGateway.namespace` default. Set this to the same namespace, as above. Get it wrong and nothing reports an error. The agent starts and serves requests while every span batch fails inside it with `Failed to resolve '…-gw-gateway-gateway-runtime.default-default' ([Errno -2] Name or service not known)`, so the Traces view stays empty; and the backend agent routes forward to has no reachable host, so invocation never reaches the gateway. The only evidence is a `Transient error` warning in the agent's own log. If you install the gateway extension into `-` instead, leave this unset. Container registry configuration The chart defaults are configured for a local k3d cluster with an in-cluster registry at `host.k3d.internal:10082`. For other environments, override the registry settings: ```bash # Example: external registry with a base domain helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set global.baseDomain="yourdomain.com" \ --set global.defaultResources.registry.tlsVerify=true \ --timeout 1800s # Registry endpoint will be: registry.yourdomain.com # Example: explicit registry endpoint helm install amp-platform-resources \ oci://${HELM_CHART_REGISTRY}/wso2-amp-platform-resources-extension \ --version ${VERSION} \ --namespace ${DEFAULT_NS} \ --set global.registry.endpoint="your-registry.example.com:5000" \ --set global.defaultResources.registry.tlsVerify=true \ --timeout 1800s ``` | Value | Default | Description | | -------------------------------------------- | ------------------------- | ----------------------------------------------------------- | | `global.registry.endpoint` | `host.k3d.internal:10082` | Registry endpoint for pushing images | | `global.baseDomain` | `""` | When set, registry endpoint becomes `registry.` | | `global.defaultResources.registry.tlsVerify` | `false` | Enable TLS verification for registry connections | Not every registry can back build workflows Two properties are required, and the failure surfaces only on the first agent build, long after the platform installs and verifies cleanly: * **Push-to-create.** Each build pushes its image as `-image` — a repository name that is different on every run, so repositories cannot be pre-created. The registry must create them on push. **Amazon ECR does not** and fails every build with `repository ... not found`; the same applies to any registry that requires repositories to be provisioned first. * **Static credentials.** The push step reads a fixed `.dockerconfigjson` from the `registry-push-secret` Secret (or pushes unauthenticated if it is absent). Registries whose credentials expire — ECR again, with its 12-hour tokens — cannot be refreshed by anything in the build pipeline. [CNCF Distribution](https://distribution.github.io/distribution/), Harbor (with a project set to auto-create), and GitLab's registry satisfy both. If you have no registry yet, running Distribution in-cluster behind an internal LoadBalancer with a cert-manager certificate is a few resources — but it ships with **no authentication**; put htpasswd in front of it before it carries anything real. *** ### Extensions[​](#extensions "Direct link to Extensions") These can be installed in any order after Core is ready. #### Step 5: Observability Extension (Agent Manager Observer)[​](#step-5-observability-extension-agent-manager-observer "Direct link to Step 5: Observability Extension (Agent Manager Observer)") Deploys the observer service that queries and serves trace, log, and metrics data to the console and CLI. ``` helm install amp-observability-traces \ oci://${HELM_CHART_REGISTRY}/wso2-amp-observability-extension \ --version ${VERSION} \ --namespace ${OBSERVABILITY_NS} \ --set amObserver.ocIngress.hostname="${OBS_API_PUBLIC_HOST}" \ --set amObserver.publicUrl="${OBS_API_PUBLIC_URL}" \ --set amObserver.auth.issuer="${THUNDER_PUBLIC_URL}" \ --timeout 1800s kubectl wait --for=condition=Available \ deployment/amp-observer -n ${OBSERVABILITY_NS} --timeout=600s ``` warning `amObserver.auth.issuer` must be the **public** Thunder URL, not the in-cluster service URL. The observer validates the same user token the console and `amctl` send to the Agent Manager API, so its issuer has to match `agentManagerService.config.keyManager.issuer` from Step 2. Leave it at the chart default on a custom-domain install and the traces page stays empty while the observer logs `JWT validation failed ... invalid issuer`. The chart derives `amObserver.oauth.authorizationServers` and the MCP audience entry from the values above; set them explicitly only if they must differ. #### Step 6: Evaluation Extension[​](#step-6-evaluation-extension "Direct link to Step 6: Evaluation Extension") Installs workflow templates for running automated evaluations (accuracy, safety, reasoning, tool usage) against agent traces. ``` helm install amp-evaluation-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \ --version ${VERSION} \ --namespace ${BUILD_CI_NS} \ --timeout 1800s ``` info Evaluation jobs publish scores using the `amp-publisher-client` OAuth2 credentials bootstrapped by the Thunder extension. The client secret is fetched from OpenBao at workflow runtime (`secret/amp-publisher-client-secret`) and must match the value set on `thunder.bootstrap.ampPublisherClient.clientSecret`. caution The eval job needs egress to your API server, and the NetworkPolicy matches it by address range after kube-proxy DNAT — the endpoint address, never the `kubernetes` Service ClusterIP. Check what that address is: ``` kubectl -n default get endpoints kubernetes ``` `networkPolicy.evaluationJob.apiServer.cidrs` defaults to all of RFC1918, which covers that address on most private clusters but also spans your pod and service CIDRs. Add this to the command above to narrow it to your control-plane subnet: ``` helm install amp-evaluation-extension \ oci://${HELM_CHART_REGISTRY}/wso2-amp-evaluation-extension \ --version ${VERSION} \ --namespace ${BUILD_CI_NS} \ --timeout 1800s \ --set "networkPolicy.evaluationJob.apiServer.cidrs[0]=" ``` Setting it is required if `kubernetes` resolves to a **public** endpoint (public GKE/AKS control planes) or to `100.64.0.0/10`. The default will not match those, and every evaluation will publish its scores and then report FAILED. #### Step 7: API Platform Gateway Extension[​](#step-7-api-platform-gateway-extension "Direct link to Step 7: API Platform Gateway Extension") Registers the API Platform Gateway with the Agent Manager and deploys the gateway stack. **Install this last** — it requires the Agent Manager API to be healthy and Thunder to be ready for token exchange. ``` helm install api-platform-default-default \ oci://${HELM_CHART_REGISTRY}/wso2-amp-api-platform-gateway-extension \ --version ${VERSION} \ --namespace ${DATA_PLANE_NS} \ --set agentManager.orgName=default \ --set gateway.environment=default \ --timeout 1800s kubectl wait --for=condition=complete job/api-platform-default-default-bootstrap \ -n ${DATA_PLANE_NS} --timeout=300s ``` The extension already renders an OTLP ingest route for its own gateway (`-otel-restapi` in the data-plane namespace), so nothing further is needed for the default environment. The standalone manifest below targets the per-environment namespace `-`, which does not exist until the first agent is deployed there — apply it only if you need that variant, and only after a deployment has created the namespace: ``` kubectl apply -f https://raw.githubusercontent.com/wso2/agent-manager/amp/v${VERSION}/deployments/values/otel-collector-rest-api.yaml ``` Verify ``` kubectl get apigateway api-platform-default-default -n ${DATA_PLANE_NS} # STATUS should show "Programmed" kubectl get jobs -n ${DATA_PLANE_NS} | grep api-platform-default-default-bootstrap # STATUS should show "Complete" ``` ### Provision Thunder Identity Provider for the Default Environment[​](#provision-thunder-identity-provider-for-the-default-environment "Direct link to Provision Thunder Identity Provider for the Default Environment") Every Environment needs its own dedicated Thunder ID instance. This is separate from the platform Thunder installed in Phase 1, which only handles console and API login. This instance is what issues each agent its own OAuth2 credential (AgentID) in that Environment. Agents can still be created without it, but they will never get an AgentID there — Agent Manager keeps retrying and failing in the background, and the API Platform Gateway Extension silently skips registering its ThunderKeyManager identity provider, so the Identity Providers page stays empty with no error anywhere in the platform. Run this once the Agent Manager API is reachable at `${API_PUBLIC_URL}`: ``` curl -fsSL "https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment-thunder.sh" \ -o add-environment-thunder.sh ENV_NAME=default \ DISPLAY_NAME="Default" \ ORG_NAME=default \ AMP_API_URL="${API_PUBLIC_URL}/api/v1" \ IDP_TOKEN_URL="${THUNDER_PUBLIC_URL}/oauth2/token" \ bash add-environment-thunder.sh ``` Verify ``` kubectl get pods -n amp-thunder-default-default # All pods should be Running ``` *** ## Verify and Access the Platform[​](#verify-and-access-the-platform "Direct link to Verify and Access the Platform") Run a full status check to confirm everything is running: ``` # All pods across key namespaces kubectl get pods -n openchoreo-control-plane kubectl get pods -n openchoreo-data-plane kubectl get pods -n openchoreo-workflow-plane kubectl get pods -n openchoreo-observability-plane kubectl get pods -n wso2-amp kubectl get pods -n amp-thunder # Helm releases helm list -A | grep -E 'openchoreo|amp|gateway' ``` | Service | URL | | ---------------------------- | ----------------------------------------------------- | | **Agent Manager Console** | | | **Agent Manager API** | | | **Agent Manager Observer** | | | **OTLP trace ingest (HTTP)** | | **Default credentials:** `admin` / `admin` HTTPS ingest is disabled by default on this setup, and enabling it doesn't cover the documented hosts [values-dp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-dp.yaml), installed in [Step 8](#step-8-setup-data-plane), sets `gateway.tls.enabled: false` even though it already configures `httpsPort: 19443` and a `certificateRefs` entry pointing at the `gateway-tls` Secret (created earlier by cert-manager). With TLS disabled, the Data Plane's `gateway-default` Gateway/Service only get an `http` listener — there is no 19443 listener to connect to, so an HTTPS OTLP endpoint is not reachable on a default install. Turning `gateway.tls.enabled` on would not make the OTLP host above reachable over HTTPS anyway: `gateway-tls`'s certificate only covers `*.agentmanager.localhost` and `*.am-gateway.localhost` (its configured `hostname` and SANs), not `*.gateway.localhost` — the domain `default-default.gateway.localhost` above actually uses. Reaching HTTPS on that specific host would additionally require reissuing the certificate with `*.gateway.localhost` in its `dnsNames` and pointing `gateway.tls.hostname` at it. Given that, this setup keeps TLS disabled and uses the HTTP endpoint above for trace ingestion. *** ## Cleanup[​](#cleanup "Direct link to Cleanup") Delete the entire k3d cluster and all resources: ``` k3d cluster delete amp-local ``` *** ## Production Considerations[​](#production-considerations "Direct link to Production Considerations") This installation is designed for development and exploration. For production: 1. **Identity provider** — Replace Thunder dev mode with a proper IdP (Asgardeo, Auth0, Okta) 2. **JWT authentication** — Set `IS_LOCAL_DEV_ENV=false` and configure `KEY_MANAGER_JWKS_URL` for `agent-manager-service` and `agent-manager-observer` before exposing either service; when unset, JWT signature verification is skipped and only the token payload is decoded 3. **TLS** — Replace self-signed certificates with CA-signed certificates 4. **Secrets backend** — Disable OpenBao dev mode; configure persistent storage and proper auth 5. **Observability storage** — Configure persistent volumes for OpenSearch 6. **Resource sizing** — Adjust requests/limits based on workload 7. **High availability** — Deploy multiple replicas of critical components 8. **Security hardening** — Apply network policies, RBAC, pod security standards *** ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Pods stuck in Pending Usually a resource constraint. Check node capacity: ``` kubectl describe pod -n kubectl top nodes ``` Increase Colima/Docker Desktop resources if needed. Gateway not becoming Programmed ``` kubectl logs -n openchoreo-data-plane -l app.kubernetes.io/name=gateway-operator kubectl describe apigateway api-platform-default-default -n openchoreo-data-plane ``` Plane registration issues ``` kubectl get clusterdataplane default -n default -o yaml kubectl logs -n openchoreo-control-plane -l app.kubernetes.io/name=openchoreo-control-plane ``` OpenSearch connectivity issues ``` kubectl get pods -n openchoreo-observability-plane -l app=opensearch kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \ curl -v http://opensearch.openchoreo-observability-plane.svc.cluster.local:9200 ``` Port already in use Find the process occupying the port and stop it: ``` lsof -i : ``` *** ## Reference: Configuration Files[​](#reference-configuration-files "Direct link to Reference: Configuration Files") All Agent Manager-specific configuration files used in this guide: | File | Purpose | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | [k3d-config.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/quick-start/k3d-config.yaml) | k3d cluster with all required port mappings | | [coredns-amp-custom.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/k8s/coredns-amp-custom.yaml) | CoreDNS rewrites for `*.openchoreo.localhost`, `*.amp.localhost`, `*.agentmanager.localhost`, `*.am-gateway.localhost`, `*.gateway.localhost` | | [add-environment-thunder.sh](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/scripts/add-environment-thunder.sh) | Provisions the per-environment Thunder identity provider | | [values-cp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-cp.yaml) | Control Plane — Backstage disabled, AMP Thunder OIDC | | [values-dp.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-dp.yaml) | Data Plane — gateway ports, Fluent Bit config | | [values-op.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-op.yaml) | Observability Plane — standalone OpenSearch, AMP Thunder OIDC | | [values-openbao.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/single-cluster/values-openbao.yaml) | OpenBao — dev mode, Kubernetes auth, pre-seeded secrets | | [oc-collector-configmap.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/oc-collector-configmap.yaml) | Custom OTel Collector ConfigMap for trace ingestion | | [wso2-amp-api-platform-gateway-extension](https://github.com/wso2/agent-manager/tree/amp/v0.0.0-dev/deployments/helm-charts/wso2-amp-api-platform-gateway-extension) | API Platform Gateway Helm chart (includes gateway config, bootstrap, CR) | | [otel-collector-rest-api.yaml](https://raw.githubusercontent.com/wso2/agent-manager/amp/v0.0.0-dev/deployments/values/otel-collector-rest-api.yaml) | OTel Collector REST API resource |