# Run Agent Manager on a VM with Docker Not recommended for production use Both installation paths on this page **Simple** and **Advanced** are intended for **evaluation, demos, and proof-of-concept** use only. Do not use them to run production workloads or to handle sensitive or regulated data. For production, run Agent Manager on a properly operated Kubernetes platform with high availability, a managed and backed-up database, secret management, monitoring, and a hardened, redundant ingress following your organization's production practices. Use these installers to try Agent Manager out, not to run it for real. Stronger agent isolation tiers Agents run sandboxed under the standard **runc** runtime by default. Agent Manager also supports stronger per-environment isolation tiers, **gVisor** (userspace kernel) and **Kata Containers** (per-agent VM), but they have hardware/OS requirements and need a dedicated node. For more information, see the [gVisor](/agent-manager/docs/next/guides/isolation-tiers/gvisor/.md) and [Kata Containers](/agent-manager/docs/next/guides/isolation-tiers/kata/.md) setup guides. Install Agent Manager on a Linux VM where Docker is the only host dependency. Pick the path that fits you: * **Simple**: give the installer the VM's public IP and it does everything else: hostnames are derived from the IP via [sslip.io](https://sslip.io) and TLS certificates are issued automatically by Let's Encrypt. No domain, no DNS setup, no certificate handling. Best for demos and quick evaluations. * **Advanced**: a config-file-driven installer for a **custom domain**, on a VM that is either publicly reachable or **private-network-only**. TLS is always a publicly-trusted wildcard certificate issued in-cluster by cert-manager through the ACME **DNS-01** challenge (Let's Encrypt by default, any ACME CA via `ACME_SERVER`), so issuance needs outbound access only and works on a private VM. Requires a DNS zone you control on a supported provider (Cloudflare, Route 53, Cloud DNS, or Azure DNS). Validates your config before installing and reports whether your DNS points at the VM. - Simple (IP + automatic TLS) - Advanced The simple installer exposes the platform over HTTPS using [sslip.io](https://sslip.io) hostnames derived from the VM's public IP, so there's no domain registration. ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") You need an SSH client to log into the VM. On the VM itself, install the tools below **before** running the installer. It **verifies** they are present and exits with install hints if any are missing; it does **not** install host tooling for you. The whole stack runs on Docker: k3d runs the Kubernetes cluster as Docker containers, and Caddy runs as a container. | Tool | Version | Purpose | | ---------------------------------------------------------- | ---------------------- | ------------------------ | | [Docker](https://docs.docker.com/engine/install/) | v26+ (8 GB RAM, 4 CPU) | Container runtime | | [k3d](https://k3d.io/#installation) | v5.8+ | Local Kubernetes cluster | | [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) | v1.33+ | Kubernetes CLI | | [Helm](https://helm.sh/docs/intro/install/) | v3.12+ | Package manager | `curl` and `lsof` are also required. They are usually preinstalled, but on a minimal image install them with your OS package manager (for example `sudo apt-get install -y curl lsof`). Verify everything is installed: ``` docker --version k3d --version kubectl version --client helm version --short ``` Verify the Docker daemon is running: ``` docker info > /dev/null ``` The VM also needs: * A **static (reserved) public IP** and SSH access (sudo). The install derives every hostname, TLS certificate, and OAuth issuer from the IP (`*.amp..sslip.io`), so a **changing IP breaks the install**, and stopping the VM (for example to resize its disk) releases an ephemeral IP. Reserve the address before installing. If the IP ever changes, reinstall against the new IP. * **At least 50 GB of disk.** Building and running agents pushes the in-cluster image store past 13 GB; on a smaller disk the node hits `DiskPressure`, which evicts pods and can take cluster DNS down mid-build. * **Inbound `443/tcp` open** in the cloud security group / firewall, and only 443. Certificates issue via the TLS-ALPN-01 ACME challenge, which runs inside the `:443` TLS handshake, so no inbound port 80 is ever needed. The `:443` exposure must be **TCP passthrough** (not a TLS-terminating load balancer in front), since the challenge happens inside the handshake. ## Install[​](#install "Direct link to Install") SSH into the VM and run the one-command installer with `sudo`. It downloads a single versioned install bundle (no repository clone) and runs from it: ``` # on the VM (Docker, k3d, kubectl, helm already installed) curl -fsSL https://github.com/wso2/agent-manager/releases/download/amp/v0.0.0-dev/bootstrap.sh \ | sudo bash -s -- simple \ --host \ --version 0.0.0-dev \ --email you@example.com ``` Pass `--host` the VM's **public** IPv4 address. A cloud VM usually can't read its own public IP (it's NAT'd behind the address you used to SSH in), so the installer needs it to build the `*.amp..sslip.io` hostnames. The installer runs in two phases: preflight (verify the required tools + open the firewall) and the platform install + Caddy startup. Allow 8-15 minutes. It needs `sudo` because it opens the firewall and creates the cluster. Downloading one bundle instead of cloning the repo and fetching each file separately avoids the GitHub rate-limiting that can throttle a per-file install. ### Options[​](#options "Direct link to Options") | Flag | Default | Purpose | | ------------------------ | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--host` | *(required)* | The VM's public IPv4 address | | `--version` | *(required)* | Agent Manager release to install; an existing `amp/v*` [release](https://github.com/wso2/agent-manager/releases) (the `bootstrap.sh` URL embeds the same version) | | `--email` | *(none)* | ACME contact for expiry notices | | `--no-external-gateways` | off | Drop the gateway control-plane endpoint if you won't connect external gateways | ## What gets exposed[​](#what-gets-exposed "Direct link to What gets exposed") The installer fronts the stack with [Caddy](https://caddyserver.com), an open-source web server that terminates TLS, obtains and renews Let's Encrypt certificates automatically, and reverse-proxies each public hostname to the right service. It runs as a single `amp-caddy` Docker container and is the only process listening on the internet-facing ports. Only `:443` faces the internet; all other service ports are bound to the VM's loopback and reached only by Caddy. Every public hostname resolves to the VM's IP (via sslip.io) and arrives at Caddy on `:443`; Caddy terminates TLS and reverse-proxies to the matching loopback port. The Agent Manager services are ClusterIP inside the cluster: Caddy forwards the console, API, Thunder, gateway-control-plane, and per-environment Thunder hosts to the OpenChoreo control-plane kgateway (loopback `:8080`), the observer host to the observability-plane kgateway (loopback `:11080`), and the OTel-ingest host plus the deployed-agent wildcard to the data-plane kgateway (loopback `:19080`). Each gateway routes by the preserved `Host` header. Certificates are obtained over that same `:443` using the TLS-ALPN-01 challenge, so no inbound port 80 is needed. Two of these tiers are **dynamic**: a new host appears the first time you deploy into a new org/project, and each time you create an environment. A wildcard certificate cannot be issued for them via TLS-ALPN-01 up front, so Caddy issues one concrete certificate per hostname **on demand**, at the first request to it. | URL | Purpose | | -------------------------------------------------- | ----------------------------------------------------------------------- | | `https://console.amp..sslip.io` | Console UI | | `https://api.amp..sslip.io` | Agent Manager API (used by `amctl`) | | `https://thunder.amp..sslip.io` | Thunder OAuth (login) | | `https://-.thunder..sslip.io` | Per-environment Thunder (one host per environment, created on demand) | | `https://observer.amp..sslip.io` | Agent Manager Observer | | `https://gateway.amp..sslip.io/otel` | OTel trace ingest for **externally-hosted** agents (see below) | | `https://-.agents..sslip.io/...` | Deployed-agent invocation endpoints (one wildcard host per org/project) | | `https://cp.amp..sslip.io` | Gateway control plane; connect external gateways here (on by default) | Agents deployed on this VM do not use the public OTel host `gateway.amp..sslip.io/otel` exists for agents running **outside** the platform, which set [`AMP_OTEL_ENDPOINT`](/agent-manager/docs/next/guides/amp-instrumentation/.md) themselves. Agents deployed by Agent Manager run inside the cluster and export their traces straight to the in-cluster gateway runtime instead, so their traffic never leaves the VM or passes through Caddy. ## Log in[​](#log-in "Direct link to Log in") Open `https://console.amp..sslip.io` and sign in with the seeded admin user **`admin`** (password **`admin`**). This user holds the `Agent Manager Admin` role, which grants full administrative permissions for both console and API access. The **`admin`** account is provisioned with the `Agent Manager Admin` role during bootstrap, so API calls are authenticated and authorized. Use the same credentials for both console login and API access via Bearer token authentication. ## Deployed-agent invocation[​](#deployed-agent-invocation "Direct link to Deployed-agent invocation") When you deploy an agent, its endpoint is published on a per-project host `-.agents..sslip.io` and routed by Caddy to the OpenChoreo data-plane gateway. Because these hostnames are dynamic (a new one per org/project), Caddy issues their TLS certificates **on demand** at the first request (via the same ACME challenge as the fixed hosts), rather than up front. Invocations are authenticated with a user token that the gateway validates against the public Thunder issuer. Because issuance is on demand and uses TLS-ALPN-01 (the challenge runs inside the `:443` handshake), the **very first request to a newly-deployed agent host can fail with a one-time certificate error**. That first connection is consumed by Caddy answering the ACME challenge, so the client briefly sees the challenge certificate instead of the real one. Every client hits this; only the wording differs. Chromium-based browsers (Chrome, Edge, Brave) report `ERR_CERTIFICATE_TRANSPARENCY_REQUIRED`, while other browsers and command-line clients report their own untrusted-certificate error. Issuance completes within a second or two; reload the page (or open it in a fresh tab) and it serves the trusted Let's Encrypt certificate. This only affects the first hit per new agent host; the certificate is then cached in the `amp-caddy-data` volume. The same applies to the per-environment Thunder hosts (`-.thunder..sslip.io`): they are created when you add an environment and get their certificates on demand the same way, so the first request to a brand-new environment's Thunder can show the same one-time error. amp-api advertises each agent endpoint with the `https://` scheme (the installer sets `tlsEnabled` on the service), so the console (and any other caller) invokes it over TLS directly through the wildcard site. ## TLS[​](#tls "Direct link to TLS") Caddy obtains and auto-renews trusted Let's Encrypt certificates on first start, with no manual certificate steps. Issuance uses the **TLS-ALPN-01** challenge, which runs inside the `:443` TLS handshake, so only inbound 443 is ever required and there is no port-80 dependency. Certificates and the ACME account persist in the `amp-caddy-data` Docker volume, so restarts do not re-request them. Because the challenge happens inside the TLS handshake, the public `:443` must reach Caddy as **raw TCP**; do not put a TLS-terminating load balancer in front of the VM. There is no `:80` listener, so plain `http://` URLs are not served (no automatic http→https redirect); always use the `https://` URLs the installer prints. ## Persistence and teardown[​](#persistence-and-teardown "Direct link to Persistence and teardown") Application data (PostgreSQL), issued certificates, and the k3d cluster persist across Docker/host restarts via named volumes. Everything the installer created lives either inside the k3d cluster or in the Caddy container and its two volumes, so tearing down is the three commands below, plus one optional cleanup: ``` sudo k3d cluster delete amp-local # delete the cluster (workloads + app data) sudo docker rm -f amp-caddy # remove the Caddy front door sudo docker volume rm amp-caddy-data amp-caddy-config # drop the cached certs + ACME account sudo rm -f /opt/amp/Caddyfile # optional: the generated Caddy config ``` Use `sudo`; the installer runs Docker and k3d as root. Note that the installer leaves no repository checkout on the VM, because `bootstrap.sh` unpacks the versioned bundle into a temporary directory and deletes it on exit. Teardown is therefore done with the commands above rather than an uninstall script. Deleting the cluster removes the platform's data; deleting the Caddy volumes discards the issued certificates and the ACME account, so a reinstall re-requests them from Let's Encrypt (which counts against its rate limits). ## Connect an external gateway[​](#connect-an-external-gateway "Direct link to Connect an external gateway") Agent Manager can drive external WSO2 AI gateways. The control-plane endpoint `https://cp.amp..sslip.io` is exposed by default for this. In the console, open **Infrastructure → Gateways**, generate a registration token, and follow the generated commands, which point the gateway at `cp.amp..sslip.io:443`, where it opens a control WebSocket and pulls its configuration. If you do not need external gateways, install with `--no-external-gateways` to drop this endpoint. **Security:** the registration token grants a gateway your LLM-provider API keys and proxy credentials. Treat it as a secret, revoke/regenerate it from the Gateways page when a gateway is decommissioned, and optionally restrict `cp.amp...` to known gateway source IPs at the firewall. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * **Certificates never issue / hosts unreachable from outside.** Open inbound `:443` in your cloud security group / NACL, and make sure the public `:443` reaches the VM as **raw TCP**: a TLS-terminating load balancer in front breaks the TLS-ALPN-01 challenge. The installer can't verify external reachability from inside the VM, so this surfaces as Caddy failing to obtain certificates (`docker logs amp-caddy`). * **Certificate not issued.** Check `docker logs amp-caddy`. Let's Encrypt rate limits on sslip.io are high but not infinite; if hit, retry shortly. * **Login redirect mismatch.** Confirm you reached the console via its `console.amp..sslip.io` URL, not the raw IP. * **Certificate error on first agent invocation** (`ERR_CERTIFICATE_TRANSPARENCY_REQUIRED` in Chromium-based browsers, an untrusted-certificate warning elsewhere): the per-agent certificate is issued on demand, and the first request races with that issuance. Reload the page after a second or two; it only happens once per new agent host (see [Deployed-agent invocation](#deployed-agent-invocation)). The advanced installer (`install-advanced.sh`) is config-file driven and runs **on the VM** with `sudo`. Use it when you have your own domain. It issues **publicly-trusted certificates** via cert-manager's ACME **DNS-01** challenge and terminates TLS on `:443` at a single in-cluster gateway. Because DNS-01 validates by writing a DNS TXT record rather than by connecting to the VM, the same flow works whether the VM is **public** or reachable only from a **private network**. Issuance needs outbound access only. Setup is the same for both up to and including the install; only how clients reach the VM differs. Work through [Prerequisites](#adv-prerequisites), [Configure](#adv-configure), and [Install](#adv-install), then switch the **Public network / Private network** selector below to the one matching your VM. Everything after that selector applies to both again. If you just want a quick IP-based demo with automatic TLS, prefer the **Simple** tab. ## Prerequisites[​](#adv-prerequisites "Direct link to Prerequisites") * **Docker, k3d, kubectl, Helm, curl, and `lsof`** must already be installed on the VM. The installer **verifies** they are present and exits with install hints if any are missing. It does **not** install them for you. * A **Linux VM** with at least **4 vCPUs**, **8 GB RAM**, and **50 GB of disk**, and **outbound** internet access (to pull images/charts and reach the ACME + DNS-provider APIs). Inbound `:443` is needed only for clients to reach the services, **not** for certificate issuance, so a fully private VM works. * **A domain you control**, hosted on one of the supported DNS providers: **Cloudflare**, **AWS Route 53**, **Google Cloud DNS**, or **Azure DNS**. * **A DNS-provider API credential** scoped to edit that zone's records. cert-manager uses it to write the `_acme-challenge` TXT record; you do **not** create TXT records by hand. ## Configure[​](#adv-configure "Direct link to Configure") The config file is plain shell, one `KEY=value` assignment per line. Generate an annotated template with `install-advanced.sh --init` (see [Install](#adv-install) for how to get the script), then fill in the keys below. Keep the file `chmod 600`: it holds your DNS-provider credential. | Key | Required | Purpose | | -------------------------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | | `AMP_VERSION` | yes | Agent Manager release to install (an `amp/v*` [release](https://github.com/wso2/agent-manager/releases)) | | `DOMAIN_BASE` | yes | Base domain; service hosts are derived as `.` | | `ACME_EMAIL` | yes | ACME account contact; cert-manager registers the account with it | | `DNS_PROVIDER` | yes | `cloudflare`, `route53`, `clouddns`, or `azuredns`; supply that provider's credentials below | | `ACME_SERVER` | no | Override the ACME directory (e.g. Let's Encrypt **staging**) while testing, to avoid rate limits | | `EXTERNAL_GATEWAYS` | no | `true` (default) exposes the `cp` endpoint for external data-plane gateways | | `HOST_CONSOLE`, `HOST_API`, `HOST_THUNDER`, `HOST_OBSERVER`, `HOST_GATEWAY`, `HOST_CP` | no | Override an individual service hostname (default `.`) | | `AGENTS_BASE` | no | Base for deployed-agent hostnames (default `agents.`) | With `DOMAIN_BASE=amp.mycompany.com`, the derived hosts are `console.amp.mycompany.com`, `api.amp.mycompany.com`, `thunder.amp.mycompany.com`, `observer.amp.mycompany.com`, `gateway.amp.mycompany.com`, and `cp.amp.mycompany.com`, plus three dynamic tiers: deployed agents at `-.agents.amp.mycompany.com`, per-environment Thunder at `-.thunder.amp.mycompany.com`, and per-environment API Platform Gateways at `-.gateway.amp.mycompany.com`. cert-manager issues one **wildcard** certificate covering all of them, so there are no per-host certificate steps. ### DNS provider credentials[​](#adv-dns-provider "Direct link to DNS provider credentials") Add the block for your provider to `amp-config.env`. Only the credential is secret; keep the file `chmod 600`. Cloudflare (a scoped **Zone → DNS → Edit** API token): ``` DNS_PROVIDER=cloudflare CLOUDFLARE_API_TOKEN= ``` AWS Route 53 (IAM access key with `route53:ChangeResourceRecordSets` on the zone): ``` DNS_PROVIDER=route53 AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_REGION=us-east-1 ``` Google Cloud DNS (service-account key file readable on the VM): ``` DNS_PROVIDER=clouddns GCP_PROJECT=my-gcp-project GCP_SERVICE_ACCOUNT_FILE=/opt/amp/gcp-dns-sa.json ``` Azure DNS (service principal): ``` DNS_PROVIDER=azuredns AZURE_TENANT_ID= AZURE_CLIENT_ID= AZURE_CLIENT_SECRET= AZURE_SUBSCRIPTION_ID= AZURE_RESOURCE_GROUP= ``` ## How TLS and routing work[​](#adv-tls "Direct link to How TLS and routing work") cert-manager (installed in the cluster) runs the ACME DNS-01 challenge, issues one **wildcard certificate** into a Kubernetes Secret, and auto-renews it. Routing uses a **front-proxy** model. A single consolidated Gateway (`amp-consolidated-gateway`) is the only thing listening on `:443`; it terminates TLS with the wildcard certificate and then forwards **by `Host` header** to each plane's own `gateway-default` gateway, which keeps its native routes. Three HTTPRoutes do the forwarding, and because two of the planes live in other namespaces, each cross-namespace hop is authorised by a ReferenceGrant. The wildcard routes matter because those tiers are **dynamic**: deploying into a new org/project adds a `-.agents.` host (agents in an existing project share theirs), and each environment you create adds both a `-.thunder.` host and an `-.gateway.` host for its API Platform Gateway. `*.agents`, `*.thunder`, and `*.gateway` are matched by the front-proxy routes and covered by the wildcard certificate up front, so nothing has to be re-issued or re-attached after install. Note that `gateway./otel` carries traces only from agents hosted **outside** the platform, which set [`AMP_OTEL_ENDPOINT`](/agent-manager/docs/next/guides/amp-instrumentation/.md) themselves. Agents deployed by Agent Manager export straight to the in-cluster gateway runtime, so their traces never reach the `:443` listener at all. ## Install[​](#adv-install "Direct link to Install") Download and unpack the versioned install bundle, then generate an annotated config template with `--init`: ``` # on the VM VER= # e.g. an amp/v* release tag, without the "amp/v" prefix curl -fsSL "https://github.com/wso2/agent-manager/releases/download/amp/v${VER}/amp-vm-bundle-${VER}.tar.gz" | tar xz deployments/vm/install-advanced.sh --init > amp-config.env chmod 600 amp-config.env ``` Edit `amp-config.env` with the keys from [Configure](#adv-configure), then validate and preview without touching the cluster: ``` sudo deployments/vm/install-advanced.sh --config amp-config.env --dry-run ``` The dry run loads the config, runs the advisory DNS check, and prints the derived hosts, the wildcard certificate's SAN list, the helm overrides, and every cert-manager and gateway resource it would apply. It never contacts the cluster and it redacts the DNS-provider credential. When it looks right, run the real install: ``` sudo deployments/vm/install-advanced.sh --config amp-config.env ``` The installer runs in three phases: preflight (verify tools + firewall), platform install, and TLS (cert-manager issues the wildcard certificate via DNS-01, then the consolidated gateway serves `:443`). Allow 15-20 minutes plus a few minutes for issuance. It needs `sudo` because it opens the firewall and creates the cluster. On completion it prints the access URLs. There are **no per-setting command-line flags** on the advanced path; everything comes from the config file. Use Let's Encrypt staging first Set `ACME_SERVER=https://acme-staging-v02.api.letsencrypt.org/directory` for your first run. Staging exercises the entire DNS-01 flow but has far looser rate limits than production, which enforces a weekly cap per registered domain that is easy to burn through while iterating. The staging certificate is **not** publicly trusted, so browsers and `curl` will warn, which is expected. Once a staging run succeeds end to end, remove `ACME_SERVER` (or point it at production), delete the cluster, and reinstall for a trusted certificate. If you opened the console in a browser while the staging certificate was being served and clicked through the warning, clear that decision after switching to production. Browsers remember the approval for the rest of the session and keep reporting the page as insecure even once the trusted certificate is in place. Chrome shows "active content with certificate errors" while simultaneously reporting the certificate itself as valid and trusted. Quit **every** window of that browser profile (for an incognito window, closing one tab is not enough, because the session survives until all incognito windows are closed) and reopen it. If you prefer a single command and already have `amp-config.env`, `bootstrap.sh` downloads the bundle and runs the same installer for you: ``` curl -fsSL "https://github.com/wso2/agent-manager/releases/download/amp/v${VER}/bootstrap.sh" \ | sudo bash -s -- advanced --config amp-config.env ``` Note that `bootstrap.sh` cannot emit the template, because it needs a version before it can download the bundle. Use the `--init` flow above to create the config. ## How clients reach the VM[​](#adv-reachability "Direct link to How clients reach the VM") The install is identical either way; what differs is DNS and how you get a browser to the console. Pick the one that matches your VM. The sections after this apply to both. * Public network * Private network Use this when the VM has a public IP and clients reach it directly over the internet. Open inbound **`443/tcp` only** in your cloud security group or firewall. Nothing else needs to be reachable: certificate issuance is egress-only, and every service is multiplexed behind the single `:443` listener by `Host` header. ### DNS records[​](#adv-dns-records "Direct link to DNS records") Certificate issuance itself needs **no A records**, because cert-manager proves control of the zone by writing the `_acme-challenge` TXT record through your provider credential. A records exist only so clients can **reach** the services. Point them at the VM's public IP: ``` *.amp.mycompany.com A # console/api/thunder/observer/gateway/cp thunder.amp.mycompany.com A # explicit, see the warning below *.thunder.amp.mycompany.com A # per-environment Thunder gateway.amp.mycompany.com A # explicit, see the warning below *.gateway.amp.mycompany.com A # per-environment API Platform Gateways *.agents.amp.mycompany.com A # deployed agents ``` `thunder.` and `gateway.` need their own explicit A records A DNS wildcard never matches a name that already exists as a node in the zone ([RFC 4592](https://datatracker.ietf.org/doc/html/rfc4592#section-2.2.1)). Creating `*.thunder.amp.mycompany.com` brings `thunder.amp.mycompany.com` into existence as an *empty non-terminal*, which immediately stops `*.amp.mycompany.com` from covering it. Without the explicit record, `thunder.amp.mycompany.com` resolves to nothing and **OAuth login breaks**, even though every other host works. `*.gateway.amp.mycompany.com` does exactly the same thing to `gateway.amp.mycompany.com`, which serves OTel ingest for externally-hosted agents. Add the explicit A record whenever you add either deeper wildcard. Verify every tier resolves before installing. The `x-y.` names are throwaway probes: they resolve only if the wildcard above them exists. ``` for h in console.amp.mycompany.com thunder.amp.mycompany.com \ gateway.amp.mycompany.com x-y.agents.amp.mycompany.com \ x-y.thunder.amp.mycompany.com x-y.gateway.amp.mycompany.com; do printf '%-40s %s\n' "$h" "$(dig +short "$h" A)" done ``` Use this when the VM has **no public IP** and is reachable only from inside your network. The advanced installer supports this unchanged, because DNS-01 validation never requires an inbound connection: the ACME CA reads a TXT record from your DNS provider instead of calling the VM. The result is a **publicly-trusted certificate on a private host**, with no private CA to distribute. The VM still needs **outbound** internet access to pull images and charts and to reach the ACME and DNS-provider APIs. On Google Cloud, for example, that means a **Cloud NAT** gateway attached to the VM's subnet; AWS and Azure have equivalent NAT gateways. ### Reaching the console[​](#reaching-the-console "Direct link to Reaching the console") You have two options. If your network has private DNS and routing to the VM, add the same four records listed under **Public network** but pointing at the VM's **private** IP, and connect normally. A public zone may hold private A records, and split-horizon DNS is fine too. Be aware that publishing private addresses in a **public** zone often does not survive the trip to the client. Many resolvers (home routers, corporate DNS, and public resolvers alike) drop RFC-1918 answers for public domains as DNS-rebinding protection, returning an empty result even though the record is correct. The symptom is a name that resolves from the authoritative nameserver but not from the client, and the browser reports `ERR_NAME_NOT_RESOLVED`. Compare the two before assuming the record is wrong: ``` dig +short A # your resolver dig +short @ A # the zone itself ``` If they disagree, use the tunnel plus `/etc/hosts` approach below rather than relying on DNS. Otherwise, tunnel to the VM and resolve the hostnames locally. On Google Cloud, for example, Identity-Aware Proxy forwards TCP without giving the VM a public address: Allow IAP to reach `:443`. The stock `default-allow-https` rule only applies to instances tagged `https-server`, so an untagged VM needs an explicit rule for the IAP source range: ``` gcloud compute firewall-rules create allow-iap-https \ --direction=INGRESS --action=allow --rules=tcp:443 \ --source-ranges=35.235.240.0/20 ``` Open the tunnel, binding it to the same port the gateway serves so the hostnames work unmodified. Port 443 is privileged, so on Linux and macOS this needs `sudo` (or a one-off `setcap`/`authbind` grant for the `gcloud` Python interpreter); without it the tunnel exits with a permission error on bind: ``` sudo gcloud compute start-iap-tunnel 443 --local-host-port=localhost:443 ``` Then point every service hostname at the tunnel in your workstation's `/etc/hosts`: ``` 127.0.0.1 console.amp.mycompany.com api.amp.mycompany.com thunder.amp.mycompany.com 127.0.0.1 observer.amp.mycompany.com gateway.amp.mycompany.com cp.amp.mycompany.com ``` Deployed agents need their own `/etc/hosts` entries `/etc/hosts` has no wildcard support, so the `*.agents`, `*.thunder`, and `*.gateway` tiers are not covered by the lines above. Each time you deploy an agent or create an environment, add its exact hostname: ``` 127.0.0.1 -.agents.amp.mycompany.com 127.0.0.1 -.thunder.amp.mycompany.com 127.0.0.1 -.gateway.amp.mycompany.com ``` If agent invocation fails with a DNS error while the console works, this is almost always why. A local DNS resolver that supports wildcards (dnsmasq, or a private DNS zone) avoids the per-agent edits. TLS still validates normally through the tunnel: the certificate is publicly trusted and the `Host` header is preserved, so the browser sees the real hostname and the gateway routes on it. ## Persistence and teardown[​](#adv-persistence "Direct link to Persistence and teardown") Application data (PostgreSQL), the issued certificate (a Kubernetes Secret, auto-renewed by cert-manager), and the k3d cluster persist across restarts. To tear down completely, delete the cluster. Everything the installer created lives inside it, including the certificate and the gateway resources: ``` sudo k3d cluster delete amp-local # delete the cluster (workloads, app data, cert, gateways) ``` To remove only the TLS and routing layer while keeping the platform running (for example to re-issue against a different ACME server after a staging run), delete the resources the installer applies in its third phase: ``` kubectl delete httproute amp-frontproxy-controlplane amp-frontproxy-observability \ amp-frontproxy-dataplane -n openchoreo-control-plane kubectl delete gateway amp-consolidated-gateway -n openchoreo-control-plane kubectl delete certificate amp-wildcard-tls -n openchoreo-control-plane kubectl delete secret amp-wildcard-tls -n openchoreo-control-plane kubectl delete referencegrant amp-frontproxy-to-services -n openchoreo-observability-plane kubectl delete referencegrant amp-frontproxy-to-services -n openchoreo-data-plane kubectl delete clusterissuer amp-acme-dns01 kubectl delete secret amp-dns01-credentials amp-acme-dns01-account-key -n cert-manager ``` Delete the `amp-wildcard-tls` Secret as well as the Certificate. cert-manager will not re-issue into a Secret that already holds a valid certificate, so leaving it behind silently keeps the old (for example, staging) certificate in place. The ACME account key Secret only needs deleting if you are also changing `ACME_EMAIL` or the ACME server. **Changing the domain or hostnames after install requires a full teardown first.** The platform install is idempotent in the "create if missing" sense, so editing `DOMAIN_BASE` (or the `HOST_*` overrides) and re-running does **not** reconfigure already-installed services. To move an existing install to a different domain, delete the cluster and install again with the new config. ## Connect an external gateway[​](#adv-external-gw "Direct link to Connect an external gateway") The control-plane endpoint `https://cp.` is exposed by default. Generate a registration token from **Infrastructure → Gateways** and follow the generated commands. Set `EXTERNAL_GATEWAYS=false` to drop the endpoint if you do not connect external gateways. The registration token grants a gateway your LLM-provider API keys, so treat it as a secret and revoke it when a gateway is decommissioned. ## Troubleshooting[​](#adv-troubleshooting "Direct link to Troubleshooting") * **Config rejected before install.** The installer names the missing/invalid key (e.g. an unsupported `DNS_PROVIDER`, or a missing provider credential). Fix `amp-config.env` and re-run. * **Certificate never becomes Ready.** cert-manager could not complete the DNS-01 challenge. Inspect it with `kubectl describe certificate amp-wildcard-tls -n openchoreo-control-plane` and `kubectl get challenge -A`. Common causes: the provider credential can't write the zone, the zone isn't delegated to that provider, or a production rate limit. Re-run with `ACME_SERVER` pointed at Let's Encrypt staging to iterate. * **Cert issued but a host is unreachable.** Confirm the service host's A record points at the VM (public or private IP) and that clients can reach `:443` on it. Issuance does not need this, but client access does. * **Login fails but every other host works.** `thunder.` is probably not resolving because the deeper `*.thunder` wildcard shadows it. Check with `dig +short thunder. A` and add the explicit A record (see **DNS records** under [How clients reach the VM](#adv-reachability)). * **A host returns a connection failure rather than an HTTP status.** The front-proxy route for that plane is not bound. List them with `kubectl get httproute -n openchoreo-control-plane` and check that each `amp-frontproxy-*` route reports `Accepted=True` and `ResolvedRefs=True`; a failed `ResolvedRefs` on the observability or data-plane route usually means its ReferenceGrant is missing. Note that a gateway-level `404` is a *healthy* response from an unrouted path: it proves TLS terminated and the request reached the gateway. * **Deployed agent unreachable while the console works.** On a public VM, confirm the `*.agents.` A record exists; on a private VM, add the agent's exact hostname to `/etc/hosts` (see the **Private network** tab under [How clients reach the VM](#adv-reachability)), since `/etc/hosts` cannot match wildcards. * **Certificate is untrusted after switching to production.** cert-manager will not re-issue into a Secret that still holds a valid staging certificate. Delete the Certificate and its Secret so it re-issues (see [Persistence and teardown](#adv-persistence)). * **Changed the domain but the console still shows the old hostnames.** Re-running with a new `DOMAIN_BASE` does not reconfigure existing releases. Delete the cluster and reinstall (see [Persistence and teardown](#adv-persistence)).