Skip to main content
Copy page
Version: Cloud

Observability

WSO2 Agent Manager provides full-stack observability for AI agents — whether they are deployed through the platform or running externally. Traces, metrics, and logs flow into a centralized store that you can query and analyze through the AMP Console.

Overview​

Observability in AMP is built on OpenTelemetry, the industry-standard framework for distributed tracing and instrumentation. Every agent interaction — LLM calls, tool invocations, MCP requests, retrieval operations, and agent reasoning steps — is captured as a structured trace and stored for analysis.

There are two places an agent can run, and two ways to instrument it. AMP supports any combination of the two:

  • Where it runs: platform-hosted (deployed through AMP) or externally-hosted (anywhere else — local, on-prem, third-party cloud).
  • How it's instrumented: auto-instrumentation (zero-code, for frameworks the Traceloop SDK covers) or manual instrumentation (you emit spans yourself against AMP's published contract).

Platform-Hosted Agents​

When you deploy a Python agent through WSO2 Agent Manager with auto-instrumentation enabled, AMP injects a pre-built init container that pre-installs the Traceloop SDK into the agent's Python environment. No code changes are required.

Picking an AMP instrumentation version​

When you create a Python agent in the Console with auto-instrumentation on, you select an AMP instrumentation version. Each version pins a specific Traceloop SDK version, so a deployment is reproducible and existing agents don't move when the platform default changes. The selector is pre-filled with the platform default; most users never change it.

You also pick the agent's Python version from a fixed dropdown (currently 3.10, 3.11, 3.12, 3.13) — the init container's Python must match the agent's runtime, so the dropdown is constrained to the set AMP has built images for. See the AMP instrumentation version mapping for the current matrix.

What Gets Instrumented​

The Traceloop SDK instruments a wide range of AI frameworks automatically:

CategoryExamples
LLM providersOpenAI, Anthropic, Azure OpenAI
Agent frameworksLangChain, LlamaIndex, CrewAI, Haystack
Vector storesPinecone, Weaviate, Chroma, Qdrant
MCP clientsAny MCP tool calls made by the agent

Trace Attributes Captured​

Each span is enriched with metadata that makes it possible to evaluate and debug agent behaviour:

  • LLM spans: model name, prompt tokens, completion tokens, latency, finish reason
  • Tool spans: tool name, input arguments, output, execution time
  • Agent spans: agent name, step number, reasoning output
  • Root span: agent ID, deployment ID, correlation ID, end-to-end latency

Externally-Hosted Agents​

Agents that are not deployed through AMP — for example, agents running locally, on-premises, or in a third-party cloud — can still send traces to AMP. These are called Externally-Hosted Agents.

Registration​

  1. In the AMP Console, open your Project and click + Add Agent.
  2. Choose Externally-Hosted Agent.
  3. Provide a Name and optional description, then click Register.
  4. The Setup Agent panel opens automatically with a Zero-code Instrumentation Guide.

Install the Package​

pip install amp-instrumentation

Generate an API Key​

In the Setup Agent panel, select a Token Duration and click Generate. Copy the key immediately — it will not be shown again.

Set Environment Variables​

export AMP_OTEL_ENDPOINT="http://localhost:22893/otel"
export AMP_AGENT_API_KEY="<your-generated-api-key>"

Run with Instrumentation​

Wrap your agent's start command with amp-instrument:

amp-instrument python my_agent.py
amp-instrument uvicorn app:main --reload
amp-instrument poetry run python agent.py

No changes to your agent code are required. The same Traceloop-based auto-instrumentation applies — all supported AI frameworks are traced automatically.

Manual Instrumentation​

If your agent runs on a framework the Traceloop SDK doesn't cover well, or you want full control over the spans you emit, turn auto-instrumentation off and emit your own spans against AMP's published contract.

This works for both deployment models:

  • Platform-hosted agents with auto-instrumentation disabled: AMP's env-injection trait still sets AMP_OTEL_ENDPOINT and AMP_AGENT_API_KEY for you. You're responsible for emitting the spans.
  • Externally-hosted agents: install amp-instrumentation for the init_otel() helper, set the two environment variables yourself, and emit spans from your own code.

The contract is layered on the OpenTelemetry GenAI semantic conventions (gen_ai.* + db.*), with a few traceloop.* extension keys for the gaps OTel hasn't standardized yet. See the manual instrumentation reference on the amp-instrumentation page for the full attribute table, transport details, and a worked init_otel() example.

Trace Visibility in AMP Console​

Once traces start flowing in, you can explore them in the AMP Console under your agent's sidebar:

  • OBSERVABILITY → Traces — search and inspect individual traces by time range or correlation ID; expand a trace to see LLM spans, tool spans, and agent reasoning steps