Skip to main content
Copy page
Version: Next

Configure LLM Providers for an Agent

Agents can be configured to use one or more LLM Service Providers registered at the organization level. The configuration process differs slightly between Platform-hosted and External agents, but both follow the same pattern: attach an org-level provider to the agent with optional guardrails.

Prerequisites​

  • At least one LLM Service Provider registered at the org level (see Register an LLM Service Provider)
  • An agent created in a project (Platform-hosted or External)

Overview: Agent Types​

TypeDescription
PlatformAgent code is built and deployed by the platform from a GitHub repository. The platform injects LLM credentials as environment variables.
ExternalAgent is deployed and managed externally. The platform registers it and provides the gateway endpoint URL + API key for the LLM provider.

Configuring LLM for a Platform-Hosted Agent​

Step 1: Open the Agent​

  1. Navigate to your project (Projects → select project → Agents).
  2. Click on a Platform-tagged agent.
  3. In the left sidebar, click Configure.

Step 2: Add an LLM Configuration​

The Configure page displays the LLM Configurations section listing all LLM configurations currently attached to this agent.

  1. Click + Add LLM Configuration.
  2. Under Service Provider, click Select a Service Provider.
    • A side panel opens listing all org-level LLM Service Providers with their template, rate limiting status, and guardrails.
    • Select the desired provider. The panel closes automatically.
  3. Optionally, under Guardrails, click + Add Guardrail to attach guardrails specific to this agent's use of the provider.
  4. Click Save.

Note: The configuration name is auto-generated from the selected provider's template (e.g., openai, anthropic). If a name already exists, a numeric suffix is appended (e.g., openai-2).

Multiple environments: If your organization has more than one environment, tabs appear at the top of the Service Provider section — one per environment. Select a provider for each environment tab before saving. Tabs with no provider selected are marked with a warning indicator.

Step 3: Set Environment Variable Names​

After saving, the Environment Variables & Integration Guide panel opens automatically. It shows the two environment variables the platform will inject into the agent at runtime:

Variable KeyDefault Name (example)Description
urlOPENAI_URLBase URL of the LLM provider endpoint
apikeyOPENAI_API_KEYAPI key for authenticating with the LLM provider

The names are auto-generated from the provider template (e.g., OPENAI_URL, OPENAI_API_KEY for an OpenAI provider). If your agent code already reads different variable names, edit them directly in the table and click Save.

Note: Variable names are shared across all environments. The platform injects environment-specific values at runtime. If the agent is in a deployed state, the updated LLM configuration will be injected into the running pod automatically — no rebuild or redeploy is required.

Step 4: Use the Provider in Agent Code​

The panel provides two ready-to-use integration options (toggle between them):

  • Python — a code snippet using the correct SDK client for the provider template, pre-populated with the environment variable names.
  • Copilot Prompt — a prompt you can paste into an AI coding assistant (e.g., GitHub Copilot, Claude) to automatically update your code to use the injected environment variables.

Example Python snippet (OpenAI template):

import os
from openai import OpenAI

url = os.environ.get('OPENAI_URL')
apikey = os.environ.get('OPENAI_API_KEY')

client = OpenAI(
    base_url=url,
    api_key="",
    default_headers={"API-Key": apikey, "Authorization": ""}
)

You can reopen this panel at any time from the LLM configuration detail page by clicking Environment Variables & Integration Guide in the top-right corner.

Configuring LLM for an External Agent​

Step 1: Create and Register the Agent​

  1. Navigate to your project (Projects → select project → Agents).

  2. Click + Add Agent.

  3. On the Add a New Agent screen, select Externally-Hosted Agent.

    This option is for connecting an existing agent running outside the platform to enable observability and governance.

  4. Fill in the Agent Details:

    FieldDescriptionExample
    NameA unique identifier for the agentmy-external-agent
    Description (optional)Short description of what this agent doesCustomer support bot

  5. Click Register.

After registration, the agent is created with status Registered and the Setup Agent panel opens automatically.

Step 2: Instrument the Agent (Setup Agent)​

The Setup Agent panel provides a Zero-code Instrumentation Guide to connect your agent to the platform for observability (traces). Select your language from the Language dropdown (Python or Ballerina).

Python​

  1. Install the AMP instrumentation package:

    pip install amp-instrumentation

    Provides the ability to instrument your agent and export traces.

  2. Generate API Key — choose a Token Duration (default: 1 year) and click Generate. Copy the token immediately — it will not be shown again.

  3. Set environment variables:

    export AMP_OTEL_ENDPOINT="http://localhost:22893/otel"
    export AMP_AGENT_API_KEY="<your-generated-token>"

    Sets the agent endpoint and agent-specific API key so traces can be exported securely.

Ballerina​

  1. Import the Amp module in your Ballerina program:

    import ballerinax/amp as _;

  2. Add the following to Ballerina.toml:

    [build-options]
    observabilityIncluded = true

  3. Update Config.toml:

    [ballerina.observe]
    tracingEnabled = true
    tracingProvider = "amp"

  4. Generate API Key — choose a Token Duration and click Generate. Copy the token immediately.

  5. Set environment variables:

    export BAL_CONFIG_VAR_BALLERINAX_AMP_OTELENDPOINT="http://localhost:22893/otel"
    export BAL_CONFIG_VAR_BALLERINAX_AMP_APIKEY="<your-generated-token>"

You can reopen the Setup Agent panel at any time from the agent Overview page by clicking Setup Agent.

Step 3: Add an LLM Configuration​

  1. In the left sidebar, click Configure.
  2. The Configure Agent page shows the LLM Configurations section (empty for a new agent).
  3. Click + Add LLM Configuration.
  4. Under Service Provider, click Select a Service Provider.
    • A side panel opens listing all org-level LLM Service Providers, showing the template (e.g., OpenAI), deployment time, rate limiting status, and guardrails.
    • Select the desired provider.
  5. Optionally, under Guardrails, click + Add Guardrail to attach content safety policies.
  6. Click Save.

Step 4: Connect Your Agent Code to the LLM​

Immediately after saving, the Connect to LLM Provider panel opens automatically with everything needed to call the LLM from your agent code:

FieldDescription
Endpoint URLThe gateway URL for this provider — use this as the base URL in your LLM client
Header NameThe HTTP header to pass the API key (API-Key)
API KeyThe generated client key — copy it now, it will not be shown again
Example cURLA ready-to-use cURL command showing the Endpoint URL, Header Name, and API Key together

Example cURL:

curl -X POST <endpoint-url> \
  --header "API-Key: <your-api-key>" \
  -d '{"model": "", "messages": [{"role": "user", "content": "Hi..."}]}'

Configure your agent's LLM client using the Endpoint URL as the base URL and pass the API Key in the API-Key header on every request.

You can reopen the connection details panel at any time from the LLM configuration detail page by clicking Connect to LLM Provider in the top-right corner. However, the API Key is only displayed once at creation time — if lost, delete the configuration and re-add it to generate a new key.

Step 5: Run the Agent​

Run your agent.

Example: Python agent with instrumentation

amp-instrument python main.py

Managing LLM Configurations​

From the Configure Agent page, the LLM Configurations table shows all attached configurations with:

  • Name: The auto-generated name for this LLM binding (derived from the provider template, e.g., openai). Click a row to open the configuration detail page.
  • Created: When the binding was created.
  • Actions: Delete icon to remove the configuration from the agent.

Multiple configurations can be attached to a single agent, allowing the agent code to use different LLMs for different tasks by referencing their respective environment variable names (platform agents) or endpoint URLs and API keys (external agents).

Changing the Provider​

From the LLM configuration detail page, click the pencil icon on the provider card to swap the linked org-level provider. This opens the provider selection drawer. Select a new provider and click Save to apply the change.

Notes​

  • LLM provider credentials are never exposed to agent code directly — only the injected environment variables (platform agents) or the gateway endpoint + API key (external agents) are available at runtime.
  • For platform agents, if the agent is in a deployed state, the updated LLM configuration is injected into the running pod automatically — no rebuild or redeploy is required.
  • For external agents, the Endpoint URL routes traffic through the AI Gateway, enabling centralized rate limiting, access control, and guardrails configured at the org level.
  • The external agent API Key shown after saving is a one-time display — it cannot be retrieved again. If lost, delete the LLM configuration and re-add it to generate a new key.
  • The Setup Agent instrumentation step is for observability (traces) only and is independent of LLM configuration.
  • Guardrails added at the agent-LLM binding level are applied in addition to any guardrails configured on the provider itself.