Skip to content

API Platform AI Gateway

A gateway for managing and securing AI traffic, including Large Language Model (LLM) APIs and Model Context Protocol (MCP) servers.

Quick Start

  • LLM Quick Start Guide—Install the gateway, confirm the Gateway Controller admin health endpoint reports it healthy, and route traffic to an LLM provider such as OpenAI.
  • MCP Quick Start Guide—Install the gateway, confirm the Gateway Controller admin health endpoint reports it healthy, and route traffic to an MCP server.

Key Concepts

LLM Provider Template

An LLM Provider Template defines the characteristics and behaviors specific to an AI service provider, such as OpenAI, Azure OpenAI, or other LLM platforms. It describes how the gateway should interpret and extract usage and operational metadata, including prompt, completion, total, and remaining token information, as well as request and response model metadata.

Following templates are shipped out-of-the-box

  • OpenAI
  • Azure OpenAI
  • Anthropic
  • AWS Bedrock
  • Azure AI Foundry
  • Gemini

LLM Provider

An LLM Provider represents a connection to an AI backend service such as OpenAI, Azure OpenAI, or other LLM APIs. Platform administrators configure LLM Providers to define:

  • The LLM Provider Template
  • The upstream LLM service URL
  • Authentication credentials (API keys, tokens)
  • Access control rules for which endpoints are exposed
  • Budget control policies, such as token-based rate limiting
  • Organization-wide policies such as guardrails

Once configured, the LLM Provider allows traffic to flow through the gateway to the AI backend.

To connect the gateway to AWS Bedrock, see Configure an AWS Bedrock LLM Provider. The guide covers both Bedrock bearer API keys and AWS Signature Version 4 (SigV4) authentication.

LLM Proxy

An LLM Proxy allows developers to create custom API endpoints that consume an LLM Provider, while inheriting administrator-enforced access control, budgeting and organization-wide policies defined at the provider level. Each proxy gets its own URL context (e.g., /assistant) and can have its own policies applied. This enables:

  • Multiple AI applications to share a single LLM Provider
  • A single OpenAI-compatible endpoint to route requests to multiple LLM providers. See Multi-Provider Routing for LLM Proxies.
  • Per-application policies such as prompt management and guardrails
  • Separation between platform administration and application development

MCP Proxy

An MCP Proxy routes Model Context Protocol traffic to MCP servers. MCP is a protocol that enables AI assistants to interact with external tools and data sources. With MCP Proxies, you can:

  • Expose MCP servers through a centralized gateway
  • Apply authentication and access control to MCP traffic
  • Manage multiple MCP servers from a single control plane

Streaming

When an upstream service streams its response, the gateway relays it to the client chunk by chunk instead of buffering the whole response. This holds for LLM providers and LLM proxies, and needs no configuration. On MCP proxies, request bodies stream, but response bodies stay buffered. See Real-time AI streaming.

Default Ports

Port Service Description
8080 Router HTTP traffic
8443 Router HTTPS traffic
9090 Gateway-Controller REST API
9094 Gateway-Controller Admin Health and admin endpoints

Architecture

                           ┌─────────────────┐
                           │ LLM Providers   │
                           │ (OpenAI, etc.)  │
                           └────────▲────────┘
                                    │
┌──────────┐    ┌──────────────┐    │
│ AI Apps  │───▶│  AI Gateway  │────┤
└──────────┘    └──────────────┘    │
                                    │
                           ┌────────▼────────┐
                           │  MCP Servers    │
                           └─────────────────┘

How it works:

  1. Administrators verify the Gateway-Controller admin health endpoint and configure LLM Providers and MCP Proxies via the Gateway-Controller API
  2. Developers create LLM Proxies to build AI applications on top of available providers
  3. The gateway routes traffic, applies policies, and manages authentication

AI Guardrails

AI Guardrails allow you to enforce safety, content, and compliance policies on AI traffic flowing through the AI Gateway. They can be applied at the LLM Provider level (organization-wide), at the LLM Proxy level (per-application), or on MCP Proxies.

The complete and up-to-date guardrail catalogue — with configuration references and examples — is maintained in the gateway-controllers repository: gateway-controllers documentation

You can extend the AI Gateway with custom guardrail policies by building a custom gateway image using the ap CLI. See Customizing the Gateway by Adding and Removing Policies.

Documentation

The following table lists the AI Gateway documentation sections and what each one covers:

Section Description
LLM LLM provider configuration, guardrails, prompt management, and semantic caching
MCP MCP proxy setup and policies
Real-time AI streaming Streamed responses across providers and proxies, and how policies and analytics behave
Observability Logging and tracing configuration
Analytics Analytics integrations (Moesif)
Policies and Guardrails Gateway policies and guardrails for AI traffic control
Gateway Controller Management API REST API reference for managing LLM providers, LLM proxies, MCP proxies, certificates, and secrets
Production deployment High-availability Kubernetes deployment with Helm, an external database, replicated workloads, and AI workload tuning
AI Workspace The control plane for governing LLM providers, proxies, and policies across every gateway you run