Skip to content

Custom provider

Connect the AI Gateway to an LLM service that none of the templates the gateway ships covers. You end up with a template of your own that tells the gateway how to read that service's responses, and an LLM Provider that uses it.

If the service you're connecting does have a shipped template, use that provider's page instead. Provider templates lists the seven the gateway ships.

This page is for platform administrators, who hold the upstream credentials.

What you need from your provider

Collect these four things before you define the template:

Value Where it comes from
Endpoint URL Your provider's API documentation.
Auth header and scheme Your provider's API documentation.
API key or credential Your account with the provider.
Token and model locations A sample response from the provider. You need to know where it reports prompt, completion and total token counts, and where it names the model.

The last row is the one a custom provider adds. For the seven shipped templates the gateway already knows these locations; for yours, you supply them.

Define the template

A template tells the gateway where to find usage metadata in your provider's traffic. Each value is read from one of four locations:

  • payload: Extract from JSON response body using JSONPath expressions (e.g., $.usage.prompt_tokens)
  • header: Extract from HTTP response headers using header name (e.g., x-ratelimit-remaining-tokens)
  • queryParam: Extract from a URL query parameter
  • pathParam: Extract from URL path using regular expressions (e.g., (?<=models/)[a-zA-Z0-9.\-]+)

An LlmProviderTemplate takes this structure:

apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProviderTemplate
metadata:
  name: <template-id>
spec:
  displayName: <Display Name>
  groupId: <template-family-id>
  managedBy: <template-owner>
  version: <template-version>
  # Provider characteristics
  promptTokens:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>
  completionTokens:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>
  totalTokens:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>
  remainingTokens:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>
  requestModel:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>
  responseModel:
    location: <payload|header|queryParam|pathParam>
    identifier: <extraction-pattern>

Only metadata.name and spec.displayName are required, so define the extraction values your provider reports and omit the rest. For every field, including the resourceMappings block that overrides extraction paths for one resource, see Provider templates.

Deploy the template

Deploy the template through the management API before you create a provider that names it:

curl -X POST http://localhost:9090/api/management/v1/llm-provider-templates \
  -H "Content-Type: application/yaml" \
  -u "$ADMIN_USERNAME:$ADMIN_PASSWORD" \
  --data-binary @- <<'EOF'
apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProviderTemplate
metadata:
  name: custom-provider
spec:
  displayName: Custom Provider
  groupId: custom-provider
  managedBy: customer
  version: v1.0
  totalTokens:
    location: payload
    identifier: $.tokens.total
EOF

The value of metadata.name becomes the template ID. This example deploys a template with the ID custom-provider, which reads a total token count from $.tokens.total in the response payload.

Configure the provider

Deploy the provider through the management API, following the procedure in Create and configure an LLM provider. A provider built on a custom template takes this shape:

apiVersion: gateway.api-platform.wso2.com/v1
kind: LlmProvider
metadata:
  name: custom-provider-instance
spec:
  displayName: Custom Provider
  version: v1.0
  template: custom-provider
  context: /providers/custom
  upstream:
    url: <provider-endpoint>
    auth:
      type: api-key
      header: <provider-auth-header>
      value: <provider-api-key>
  accessControl:
    mode: deny_all
    exceptions:
      - path: <chat-completions-path>
        methods: [POST]

The template field names the template ID you deployed. Replace the four placeholders with the values you collected:

  • <provider-endpoint> — the base URL from your provider's API documentation.
  • <provider-auth-header> — the header name that documentation specifies for authentication.
  • <provider-api-key> — your credential, formatted as that documentation specifies.
  • <chat-completions-path> — the request path you expose through the gateway.

The context value sets the URL prefix the provider answers on, so this provider serves its exposed paths under /providers/custom. The accessControl block denies every upstream path except those listed as exceptions.

Manage the template

Custom templates can be updated and deleted through the same management API. The templates the gateway ships cannot be modified or deleted. For the update and delete requests, see Provider templates.