Skip to content

LLM proxy

An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., /assistant) and can have its own policies applied. This enables:

  • Multiple AI applications to share a single LLM Provider
  • A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See Multi-provider routing.
  • Per-application policies such as prompt management and guardrails
  • Separation between platform administration and application development

This page is for AI developers, who own LLM proxies. It takes you through deploying one and routing a request through it.

Who configures this

AI developers own LLM proxies. A developer creates the proxy, names the LLM provider it consumes, and attaches the policies one application needs. The access control, budgeting, and organization-wide policies set on the provider still apply.

Prerequisites

  • A running AI Gateway, with ADMIN_USERNAME and ADMIN_PASSWORD exported in the shell you run these commands from. See Install the gateway.
  • A deployed LLM provider on that gateway. A proxy names the provider it consumes in provider.id, and the gateway rejects a proxy that names one it can't find. See Create and configure an LLM provider.

Configure the proxy

The definition below deploys a proxy that consumes the openai-provider provider. Set provider.id to the metadata.name of a provider already deployed on your gateway — a value that doesn't match a deployed provider is the most common reason this request fails.

curl -X POST http://localhost:9090/api/management/v1/llm-proxies \
  -H "Content-Type: application/yaml" \
  -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \
  --data-binary @- <<'EOF'
apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProxy
metadata:
  name: openai-assistant
spec:
  displayName: OpenAI Assistant
  version: v1.0
  context: /assistant
  provider:
    id: openai-provider
  policies: []
EOF

Save the proxy definition to openai-assistant.yaml:

@'
apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProxy
metadata:
  name: openai-assistant
spec:
  displayName: OpenAI Assistant
  version: v1.0
  context: /assistant
  provider:
    id: openai-provider
  policies: []
'@ | Set-Content -Path openai-assistant.yaml -Encoding utf8

Then post it:

curl.exe -X POST http://localhost:9090/api/management/v1/llm-proxies `
  -H "Content-Type: application/yaml" `
  -u "${env:ADMIN_USERNAME}:${env:ADMIN_PASSWORD}" `
  --data-binary "@openai-assistant.yaml"

Three values shape the proxy:

  • context — the URL prefix clients call, independent of the provider's own context. This proxy answers under /assistant.
  • provider.id — the provider this proxy consumes, named by its metadata.name.
  • policies — the policies that apply to this proxy alone. An empty list deploys the proxy with none of its own; the provider's policies still apply.

Test the proxy

Send a chat completion request to the proxy's context on the gateway:

curl -X POST "https://localhost:8443/assistant/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Hi"
      }
    ]
  }' -k

curl.exe -X POST https://localhost:8443/assistant/chat/completions `
  -H "Content-Type: application/json" `
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hi"}]}' -k

The -k flag tells curl to skip Transport Layer Security (TLS) certificate verification. The router presents the self-signed listener certificate that setup.sh or setup.ps1 generates, and no certificate authority trusts it. Outside local testing, give the router a certificate from a trusted certificate authority and remove -k.

Policies

The routing policies that select a provider for a proxy, and the failover behavior that comes with them, are covered in Load balancing and failover.

Next steps