# Observability

WSO2 Agent Manager provides full-stack observability for AI agents — whether they are deployed through the platform or running externally. Traces, metrics, and logs flow into a centralized store that you can query and analyze through the AMP Console.

## Overview[​](#overview "Direct link to Overview")

Observability in AMP is built on [OpenTelemetry](https://opentelemetry.io/), the industry-standard framework for distributed tracing and instrumentation. Every agent interaction — LLM calls, tool invocations, MCP requests, retrieval operations, and agent reasoning steps — is captured as a structured trace and stored for analysis.

There are two places an agent can run, and two ways to instrument it. AMP supports any combination of the two:

* **Where it runs:** platform-hosted (deployed through AMP) or externally-hosted (anywhere else — local, on-prem, third-party cloud).
* **How it's instrumented:** auto-instrumentation (zero-code, for frameworks the Traceloop SDK covers) or manual instrumentation (you emit spans yourself against AMP's published contract).

## Platform-Hosted Agents[​](#platform-hosted-agents "Direct link to Platform-Hosted Agents")

When you deploy a Python agent through WSO2 Agent Manager with auto-instrumentation enabled, AMP injects a pre-built init container that pre-installs the Traceloop SDK into the agent's Python environment. No code changes are required.

### Picking an AMP instrumentation version[​](#picking-an-amp-instrumentation-version "Direct link to Picking an AMP instrumentation version")

When you create a Python agent in the Console with auto-instrumentation on, you select an **AMP instrumentation version**. Each version pins a specific Traceloop SDK version, so a deployment is reproducible and existing agents don't move when the platform default changes. The selector is pre-filled with the platform default; most users never change it.

You also pick the agent's **Python version** from a fixed dropdown (currently `3.10`, `3.11`, `3.12`, `3.13`) — the init container's Python must match the agent's runtime, so the dropdown is constrained to the set AMP has built images for. See the [AMP instrumentation version mapping](/agent-manager/docs/cloud/components/amp-instrumentation/.md#amp-instrumentation-version-mapping) for the current matrix.

### What Gets Instrumented[​](#what-gets-instrumented "Direct link to What Gets Instrumented")

The Traceloop SDK instruments a wide range of AI frameworks automatically:

| Category         | Examples                                |
| ---------------- | --------------------------------------- |
| LLM providers    | OpenAI, Anthropic, Azure OpenAI         |
| Agent frameworks | LangChain, LlamaIndex, CrewAI, Haystack |
| Vector stores    | Pinecone, Weaviate, Chroma, Qdrant      |
| MCP clients      | Any MCP tool calls made by the agent    |

### Trace Attributes Captured[​](#trace-attributes-captured "Direct link to Trace Attributes Captured")

Each span is enriched with metadata that makes it possible to evaluate and debug agent behaviour:

* **LLM spans**: model name, prompt tokens, completion tokens, latency, finish reason
* **Tool spans**: tool name, input arguments, output, execution time
* **Agent spans**: agent name, step number, reasoning output
* **Root span**: agent ID, deployment ID, correlation ID, end-to-end latency

## Externally-Hosted Agents[​](#externally-hosted-agents "Direct link to Externally-Hosted Agents")

Agents that are **not deployed through AMP** — for example, agents running locally, on-premises, or in a third-party cloud — can still send traces to AMP. These are called **Externally-Hosted Agents**.

### Registration[​](#registration "Direct link to Registration")

1. In the AMP Console, open your **Project** and click **+ Add Agent**.
2. Choose **Externally-Hosted Agent**.
3. Provide a **Name** and optional description, then click **Register**.
4. The **Setup Agent** panel opens automatically with a **Zero-code Instrumentation Guide**.

### Install the Package[​](#install-the-package "Direct link to Install the Package")

```
pip install amp-instrumentation
```

### Generate an API Key[​](#generate-an-api-key "Direct link to Generate an API Key")

In the Setup Agent panel, select a **Token Duration** and click **Generate**. Copy the key immediately — it will not be shown again.

### Set Environment Variables[​](#set-environment-variables "Direct link to Set Environment Variables")

```
export AMP_OTEL_ENDPOINT="http://localhost:22893/otel"
export AMP_AGENT_API_KEY="<your-generated-api-key>"
```

### Run with Instrumentation[​](#run-with-instrumentation "Direct link to Run with Instrumentation")

Wrap your agent's start command with `amp-instrument`:

```
amp-instrument python my_agent.py
amp-instrument uvicorn app:main --reload
amp-instrument poetry run python agent.py
```

No changes to your agent code are required. The same Traceloop-based auto-instrumentation applies — all supported AI frameworks are traced automatically.

## Manual Instrumentation[​](#manual-instrumentation "Direct link to Manual Instrumentation")

If your agent runs on a framework the Traceloop SDK doesn't cover well, or you want full control over the spans you emit, turn auto-instrumentation off and emit your own spans against AMP's published contract.

This works for both deployment models:

* **Platform-hosted agents** with auto-instrumentation disabled: AMP's env-injection trait still sets `AMP_OTEL_ENDPOINT` and `AMP_AGENT_API_KEY` for you. You're responsible for emitting the spans.
* **Externally-hosted agents:** install `amp-instrumentation` for the `init_otel()` helper, set the two environment variables yourself, and emit spans from your own code.

The contract is layered on the OpenTelemetry GenAI semantic conventions (`gen_ai.*` + `db.*`), with a few `traceloop.*` extension keys for the gaps OTel hasn't standardized yet. See the [manual instrumentation reference](/agent-manager/docs/cloud/components/amp-instrumentation/.md#manual-instrumentation) on the `amp-instrumentation` page for the full attribute table, transport details, and a worked `init_otel()` example.

***

## Trace Visibility in AMP Console[​](#trace-visibility-in-amp-console "Direct link to Trace Visibility in AMP Console")

Once traces start flowing in, you can explore them in the AMP Console under your agent's sidebar:

* **OBSERVABILITY → Traces** — search and inspect individual traces by time range or correlation ID; expand a trace to see LLM spans, tool spans, and agent reasoning steps