# WSO2 Agent Manager Instrumentation Package Zero-code OpenTelemetry instrumentation package for externally-hosted Python agents using the Traceloop SDK, with trace visibility in the WSO2 Agent Manager. ## Overview[​](#overview "Direct link to Overview") `amp-instrumentation` package enables zero-code instrumentation for Python agents, automatically capturing traces for LLM calls, MCP requests, and other operations. It wraps your agent's execution with OpenTelemetry tracing powered by the Traceloop SDK. For agents on a custom or non-frontier framework, or anywhere you want full control over the spans you emit, it also ships `init_otel()`, a one-line OpenTelemetry exporter setup so you can instrument the agent yourself against AMP's [manual instrumentation contract](#manual-instrumentation). ## Features[​](#features "Direct link to Features") * **Zero Code Changes**: Instrument existing applications without modifying code * **Automatic Tracing**: Traces LLM calls, MCP requests, database queries, and more * **OpenTelemetry Compatible**: Uses industry-standard OpenTelemetry protocol * **Flexible Configuration**: Configure via environment variables * **Framework Agnostic**: Works with any Python application built using a wide range of agent frameworks supported by the Traceloop SDK * **Manual path**: `init_otel()` for agents that emit their own OpenTelemetry GenAI spans ## Installation[​](#installation "Direct link to Installation") ``` pip install amp-instrumentation ``` Each release of `amp-instrumentation` pins a specific Traceloop SDK version, so a given `amp-instrumentation` version always installs a fully determined SDK. To use a different Traceloop SDK version, install a different `amp-instrumentation` version — see the [version mapping](#amp-instrumentation-version-mapping) below. ## Quick Start[​](#quick-start "Direct link to Quick Start") ### 1. Register Your Agent[​](#1-register-your-agent "Direct link to 1. Register Your Agent") First, register your agent as an external agent in the [WSO2 Agent Manager](/agent-manager/docs/v1.0.0-alpha1/getting-started/create-your-first-agent/.md) to obtain your agent API key and configuration details. ### 2. Set Required Environment Variables in terminal[​](#2-set-required-environment-variables-in-terminal "Direct link to 2. Set Required Environment Variables in terminal") Use the OTEL endpoint and agent API key obtained when you registered the agent in the previous step: ``` export AMP_OTEL_ENDPOINT="https://amp-otel-endpoint.com" # AMP OTEL endpoint export AMP_AGENT_API_KEY="your-agent-api-key" # Agent-specific key generated after registration ``` These must be shell variables, not a `.env` file `amp-instrument` reads `AMP_OTEL_ENDPOINT` and `AMP_AGENT_API_KEY` from the process environment at launch and it does not load a `.env` file. If you keep these values in a `.env` file, export them into the shell before running `amp-instrument`: ``` set -a && source .env && set +a amp-instrument python my_script.py ``` Using Pydantic Settings AMP puts extra variables in the environment at runtime (`AMP_OTEL_ENDPOINT`, `AMP_AGENT_API_KEY`). Pydantic Settings reads the whole process environment, so with its default `extra="forbid"` these undeclared variables raise an `extra_forbidden` validation error. Set `extra="ignore"` on your settings model (or scope it to your own variables with an `env_prefix`) so it leaves variables it doesn't own alone: ``` from pydantic_settings import BaseSettings, SettingsConfigDict class Settings(BaseSettings): model_config = SettingsConfigDict(extra="ignore") # your fields... ``` ### 3. Run Your Application[​](#3-run-your-application "Direct link to 3. Run Your Application") Use the `amp-instrument` command to wrap your application run command: ``` # Run a Python script amp-instrument python my_script.py # Run with uvicorn amp-instrument uvicorn app:main --reload # Run with any package manager amp-instrument poetry run python script.py amp-instrument uv run python script.py ``` That's it! Your application is now instrumented and sending traces to the WSO2 Agent Manager. What auto-instrumentation captures The Traceloop SDK generates GenAI spans by patching recognized LLM and framework SDKs (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI, and more). It does **not** trace raw HTTP calls. If your agent calls an LLM by hand with `httpx`, `requests`, or `urllib`, either call the model through a supported SDK - for example, the OpenAI SDK pointed at an OpenAI-compatible endpoint with `base_url` - or emit the spans yourself with [manual instrumentation](#manual-instrumentation). ## Manual instrumentation[​](#manual-instrumentation "Direct link to Manual instrumentation") If you run a custom or non-frontier agent framework that the Traceloop SDK doesn't cover well — or you just want full control over the spans you emit — you can instrument your agent yourself and send the spans to AMP. This section is the contract: which endpoint to send to, which header to set, and which attributes a span needs to carry to render with the full trace view in the AMP Console and feed evaluators. Spans that follow the contract render exactly the same as auto-instrumented spans. Spans that don't still appear, just without the rich view. The manual path works on both platform-hosted and externally-hosted agents. The only difference is who sets the two environment variables — AMP injects them on platform-hosted agents (via the env-injection trait when auto-instrumentation is off); you set them yourself when running externally. ### Transport[​](#transport "Direct link to Transport") * **Endpoint:** OTLP/HTTP `POST` to `${AMP_OTEL_ENDPOINT}/v1/traces`. * **Header:** `x-amp-api-key: ${AMP_AGENT_API_KEY}`. For a platform-hosted agent with auto-instrumentation disabled, AMP's env-injection trait sets `AMP_OTEL_ENDPOINT` and `AMP_AGENT_API_KEY` for you. For an externally-hosted agent, you set them yourself — `AMP_OTEL_ENDPOINT` is the AMP gateway's OTel endpoint, and `AMP_AGENT_API_KEY` is the key you generate in the Console when you register the agent. ### Setting up the exporter[​](#setting-up-the-exporter "Direct link to Setting up the exporter") `amp-instrumentation` ships an `init_otel()` helper that configures the OpenTelemetry exporter so you don't have to write the `TracerProvider` + `BatchSpanProcessor` + `OTLPSpanExporter` boilerplate yourself. It does no instrumentation; it just wires the exporter to the AMP endpoint with the API-key header. ``` import json from opentelemetry import trace from amp_instrumentation import init_otel init_otel() # reads AMP_OTEL_ENDPOINT and AMP_AGENT_API_KEY from the environment tracer = trace.get_tracer("my-agent") with tracer.start_as_current_span("chat") as span: span.set_attribute("gen_ai.operation.name", "chat") span.set_attribute("gen_ai.system", "openai") span.set_attribute("gen_ai.request.model", "gpt-4o-mini") span.set_attribute("gen_ai.request.temperature", 0.7) span.set_attribute("gen_ai.input.messages", json.dumps(input_messages)) response = call_model(...) span.set_attribute("gen_ai.response.model", response.model) span.set_attribute("gen_ai.output.messages", json.dumps(response.messages)) span.set_attribute("gen_ai.usage.input_tokens", response.usage.input_tokens) span.set_attribute("gen_ai.usage.output_tokens", response.usage.output_tokens) ``` `init_otel()` is idempotent. Calling it more than once won't double-register the tracer provider. If you'd rather not pull in `amp-instrumentation`, the same setup is about ten lines of vanilla OpenTelemetry SDK code: a `TracerProvider`, a `BatchSpanProcessor` wrapping an `OTLPSpanExporter(endpoint=/v1/traces, headers={"x-amp-api-key": })`, and `trace.set_tracer_provider(...)`. The contract is what AMP commits to, not the helper. ### The contract[​](#the-contract "Direct link to The contract") The contract is **layered**: * **Layer 1 — OpenTelemetry GenAI semantic conventions (`gen_ai.*`, plus `db.*` for retriever spans).** This is the primary set. It's a real public standard, the ecosystem is converging on it, and it covers `llm` / `embedding` / `tool` / `agent` / `retriever` spans and all their model, vendor, token, status, system-prompt, and tool data. * **Layer 2 — OpenLLMetry/Traceloop extension keys (`traceloop.*`).** Used only for the few decisions OTel hasn't standardized yet: the `chain` span kind, `rerank`, and tool-call arguments/result. These are documented, stable conventions, and they're what AMP's managed instrumentation already emits — so manual and auto spans end up identically-shaped. When OTel ratifies keys for these areas, AMP's observer will read the new standard key too, and the row will move from "OpenLLMetry ext" to "OTel GenAI" in the table below. "Required" in the table below is per span kind — it applies only when emitting that kind of span. A span missing a required key still appears in the Console; it just won't have the part of the rich view that key feeds. **Supported attributes** (full table, click to expand) | Span kind | Attribute key | Source | Required | What it enables | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | any GenAI span | `gen_ai.operation.name` (one of `chat`, `text_completion`, `embeddings`, `execute_tool`, `invoke_agent`, `create_agent`) | OTel GenAI | yes | `AmpAttributes.kind` (span icon, which card renders, which evaluators apply) | | any GenAI span | `gen_ai.system` (provider id, e.g. `openai`, `anthropic`, `aws.bedrock`, `azure.ai.openai`, `cohere`) | OTel GenAI | yes | vendor / framework chip | | any span | span `Status` set to `Error` (plus message), or `error.type` attribute | OTel (span status) | recommended | error badge on the span; error count in the trace list | | any span | W3C baggage `task_id`, `trial_id` | OTel baggage | no | joins the trace to the evaluation trial that produced it | | llm | `gen_ai.request.model` | OTel GenAI | yes | model chip; evaluator context | | llm | `gen_ai.response.model` | OTel GenAI | no | model chip (used over `request.model` when present) | | llm | `gen_ai.input.messages` (JSON array), *or* indexed `gen_ai.prompt.{i}.role` / `gen_ai.prompt.{i}.content` | OTel GenAI | yes (one form) | `AmpAttributes.input`; **LLM evaluators fail without it** | | llm | `gen_ai.output.messages` (JSON array), *or* indexed `gen_ai.completion.{i}.role` / `gen_ai.completion.{i}.content` | OTel GenAI | yes (one form) | `AmpAttributes.output`; **LLM evaluators fail without it** | | llm | `gen_ai.request.temperature` | OTel GenAI | no | temperature chip | | llm | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` (legacy `prompt_tokens` / `completion_tokens` also read) | OTel GenAI | no | token chip; trace-list token total | | llm | `gen_ai.input.tools` (JSON), *or* legacy `gen_ai.request.functions.{i}.name` / `.description` / `.parameters` | OTel GenAI / OpenLLMetry ext | no | tools accordion | | embedding | `gen_ai.request.model` (and `gen_ai.response.model`, preferred when present) | OTel GenAI | yes | model chip | | embedding | `gen_ai.usage.input_tokens` | OTel GenAI | no | token chip | | embedding | `gen_ai.prompt.{i}.content` (indexed) | OTel GenAI (de-facto) | no | `AmpAttributes.input` (the embedded text). OTel hasn't fully settled the embedding-input key; this is what's read today. | | tool | `gen_ai.tool.name` | OTel GenAI | yes | tool name header; contributes to `kind = tool` | | tool | `gen_ai.tool.description` | OTel GenAI | no | tool description (not extracted in v1) | | tool | `gen_ai.tool.call.id` | OTel GenAI | no | tool call id (not extracted in v1) | | tool | `traceloop.entity.input` / `traceloop.entity.output` (JSON) | OpenLLMetry ext | no | tool-call arguments / result. OTel has no stable key here; AMP also reads `gen_ai.input.messages` / `gen_ai.output.messages` on tool spans as a lower-priority fallback. | | agent | `gen_ai.agent.name` | OTel GenAI | yes | agent name; `kind = agent` | | agent | `gen_ai.agent.description` | OTel GenAI | no | agent description (not extracted in v1) | | agent | `gen_ai.agent.tools` (JSON) | OTel GenAI | no | tools accordion | | agent | `gen_ai.request.model` | OTel GenAI | no | model chip | | agent | `gen_ai.system` | OTel GenAI | no | framework chip | | agent | `gen_ai.system_instructions` (or `gen_ai.prompt.0.content` with role `system`) | OTel GenAI | no | system-prompt card | | agent | `gen_ai.conversation.id` | OTel GenAI | no | conversation grouping | | agent | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | OTel GenAI | no | token chip; trace-list total | | agent | `gen_ai.input.messages` / `gen_ai.output.messages` (or the indexed form) | OTel GenAI | recommended | `AmpAttributes.input` / `output`; agent-level evaluators need it | | retriever | `db.system.name` (`pinecone`, `chroma`, `qdrant`, `weaviate`, `milvus`, `pgvector`, …) | OTel DB semconv | yes | `kind = retriever`; vectorDB chip | | retriever | `db.collection.name` | OTel DB semconv | no | collection chip | | retriever | `db.vector.query.top_k` | OTel DB semconv | no | Top-K chip. Extracted as int, float, or string. | | chain / workflow | `traceloop.span.kind` = `workflow` or `task` | OpenLLMetry ext | no | `kind = chain` (the chain icon). OTel has no signal for this; without it the span renders as a plain span. | | chain / workflow | `traceloop.entity.input` / `traceloop.entity.output` (JSON) | OpenLLMetry ext | no | chain I/O | | rerank | any of: `traceloop.span.kind` = `rerank`; `gen_ai.operation.name` = `rerank` / `reranking`; `rerank.model`; a `gen_ai.request.model` like `rerank-english-*` / `rerank-multilingual-*`; or a span named `rerank` / `reranker` | OpenLLMetry ext / de-facto (not standard OTel) | no | `kind = rerank` (the rerank icon, nothing more — see below) | Anything not on this list is ignored. ### Partially supported kinds[​](#partially-supported-kinds "Direct link to Partially supported kinds") A couple of kinds work as a *kind* but don't have their full data card extracted yet: * **Rerank** is recognised as a kind only — a rerank span gets the rerank icon, no data card. There's no `RerankData` payload today and no per-kind extractor runs for rerank, so model, query, and results are not surfaced. A future enhancement will add a proper rerank payload. (Note: `rerank` isn't a value OTel GenAI enumerates for `gen_ai.operation.name`; AMP reads it because Cohere's OTel instrumentation and OpenLLMetry emit it. If OTel ratifies a rerank operation later, AMP will align.) * **Retriever documents.** The retrieved documents on a retriever span are not extracted in v1 (they aren't extracted for any instrumentation source today). The vectorDB chip, collection, and Top-K render; the document list doesn't. ### Suppressing prompt and completion content[​](#suppressing-prompt-and-completion-content "Direct link to Suppressing prompt and completion content") Prompt and completion content is captured by default. To suppress it, set `TRACELOOP_TRACE_CONTENT=false` in the agent's environment. The variable is read by the Traceloop SDK on the auto path and is documented here for parity — on a fully hand-rolled manual setup that bypasses Traceloop, you control what you put into `gen_ai.input.messages` / `gen_ai.output.messages` directly. ## Instrumenting .NET agents[​](#instrumenting-net-agents "Direct link to Instrumenting .NET agents") AMP's platform-injected auto-instrumentation covers Python (and Ballerina). A .NET agent joins AMP through the **external-agent path**: register it in the Console, then have it push spans to AMP's OTLP endpoint using the same transport and contract as above (`POST ${AMP_OTEL_ENDPOINT}/v1/traces`, header `x-amp-api-key`, `gen_ai.*` attributes). Nothing on the AMP side is .NET-specific — any process that emits contract-shaped spans to that endpoint renders in the Console and feeds evaluators. The runnable [`dotnet-agent`](https://github.com/wso2/agent-manager/tree/main/samples/dotnet-agent) sample is the worked example of everything below. ### Where the `gen_ai.*` spans come from[​](#where-the-gen_ai-spans-come-from "Direct link to where-the-gen_ai-spans-come-from") Unlike Python's Traceloop SDK, OpenTelemetry's .NET **zero-code** auto-instrumentation (the CoreCLR profiler) ships **no** GenAI instrumentation — it produces infrastructure spans (ASP.NET Core, HttpClient, database) only. In .NET, `gen_ai.*` spans are emitted by the **AI SDK itself** through an `ActivitySource`. So the primary path is SDK-native emission, and the profiler is an optional add-on for infrastructure spans. The recommended SDK is **Microsoft Agent Framework** (Microsoft's successor to Semantic Kernel and AutoGen, built on Microsoft.Extensions.AI). It emits the GenAI conventions natively — one agent invocation yields `invoke_agent` → `chat` → `execute_tool` spans, the same shape AMP's Python agents produce. ### Enabling it[​](#enabling-it "Direct link to Enabling it") Instrument the **agent** with `UseOpenTelemetry(sourceName)` and let Agent Framework auto-wire the chat-client and tool spans beneath it. Do not also instrument the chat client yourself — that disables the auto-wiring and flattens the trace. Then configure the exporter to AMP. This is the .NET analog of `init_otel()`; the sample wraps it in [`AmpTelemetry.cs`](https://github.com/wso2/agent-manager/tree/main/samples/dotnet-agent/AmpTelemetry.cs): ``` const string SourceName = "my-dotnet-agent"; var endpoint = Environment.GetEnvironmentVariable("AMP_OTEL_ENDPOINT") ?? throw new InvalidOperationException("AMP_OTEL_ENDPOINT is not set"); var apiKey = Environment.GetEnvironmentVariable("AMP_AGENT_API_KEY") ?? throw new InvalidOperationException("AMP_AGENT_API_KEY is not set"); // Exporter -> AMP. Note: when the OTLP Endpoint is set in code (rather than via // OTEL_EXPORTER_OTLP_ENDPOINT), the SDK does not append the signal path, so // include /v1/traces yourself. Sdk.CreateTracerProviderBuilder() .SetSampler(new AlwaysOnSampler()) // see note below .AddSource(SourceName) // must match the UseOpenTelemetry sourceName .AddOtlpExporter(o => { o.Endpoint = new Uri($"{endpoint.TrimEnd('/')}/v1/traces"); o.Protocol = OtlpExportProtocol.HttpProtobuf; o.Headers = $"x-amp-api-key={Uri.EscapeDataString(apiKey)}"; }) .Build(); // Agent: one UseOpenTelemetry call, same source name. AIAgent agent = new ChatClientAgent(chatClient, name: "WeatherAgent", instructions: "...", tools: [...]) .AsBuilder() .UseOpenTelemetry(sourceName: SourceName, configure: cfg => cfg.EnableSensitiveData = true) .Build(); ``` Two things that will otherwise cost you a trace with no visible error: * **`AddSource(SourceName)` must match** the `sourceName` passed to `UseOpenTelemetry`. If it doesn't, the spans are created but never exported. * **Sampling.** When the agent runs inside a web request whose ASP.NET Core activity is not itself sampled (i.e. you have not added ASP.NET Core instrumentation), the default `ParentBased` sampler drops the agent's child spans. Use `AlwaysOnSampler` (as above), or add ASP.NET Core instrumentation so the request span is recorded. ### Content capture and experimental conventions[​](#content-capture-and-experimental-conventions "Direct link to Content capture and experimental conventions") Message content (prompts, responses, tool arguments) is captured only when `EnableSensitiveData = true`. Wire it to the same `AMP_TRACE_CONTENT` toggle the rest of AMP uses so content can be suppressed without dropping spans. The .NET GenAI conventions are still experimental, so pin your package versions and set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` across your fleet to keep the span shape stable. ### Optional: infrastructure spans via zero-code[​](#optional-infrastructure-spans-via-zero-code "Direct link to Optional: infrastructure spans via zero-code") To also capture ASP.NET Core / HttpClient / DB spans around the `gen_ai.*` spans, install the OpenTelemetry .NET zero-code (CoreCLR) auto-instrumentation and set its env vars (`CORECLR_ENABLE_PROFILING=1`, `CORECLR_PROFILER={918728DD-259F-4A6A-AC2B-B85E1B658318}`, `CORECLR_PROFILER_PATH`, `DOTNET_STARTUP_HOOKS`, `OTEL_DOTNET_AUTO_HOME`), plus `OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES=` so it also exports the agent's spans. On Alpine/musl images, point `CORECLR_PROFILER_PATH` at the `linux-musl-x64` native library rather than `linux-x64`. The profiler adds infrastructure spans only; it does not itself produce `gen_ai.*` spans. ## AMP instrumentation version mapping[​](#amp-instrumentation-version-mapping "Direct link to AMP instrumentation version mapping") Each AMP-instrumentation version is a single semver shared by the PyPI package (`amp-instrumentation`) and the init-container image (`ghcr.io/wso2/amp-python-instrumentation-provider:-python`) AMP injects into platform-hosted Python agents. The version is independent of the AMP product version. One AMP-instrumentation version pins exactly one Traceloop SDK version. When AMP raises the platform default, existing agents stay on the version they were pinned to. ### Bundled baseline[​](#bundled-baseline "Direct link to Bundled baseline") These are the versions that ship with this AMP release: | AMP instrumentation version | Traceloop SDK (`traceloop-sdk`) | Supported Python versions | Init-container image tag | | --------------------------- | ------------------------------- | ------------------------- | -------------------------------------------------------------------- | | 0.4.0 (default) | 0.62.1 | 3.10, 3.11, 3.12, 3.13 | `ghcr.io/wso2/amp-python-instrumentation-provider:0.4.0-python` | | 0.3.0 | 0.61.0 | 3.10, 3.11, 3.12, 3.13 | `ghcr.io/wso2/amp-python-instrumentation-provider:0.3.0-python` | The canonical source for this table is [`.github/release-config.json`](https://github.com/wso2/agent-manager/blob/main/.github/release-config.json) under the `python-instrumentation-provider` key. Maintainers cutting a new version: see [`python-instrumentation-provider/RELEASING.md`](https://github.com/wso2/agent-manager/blob/main/python-instrumentation-provider/RELEASING.md) for the release runbook. ### Why the Console may show more than this[​](#why-the-console-may-show-more-than-this "Direct link to Why the Console may show more than this") The version set a given install actually accepts is assembled at runtime from the bundled baseline above plus any extra entries the platform admin has added through the Helm chart. So when AMP publishes, say, `0.4.0` between AMP releases, an operator can make it selectable in the Console without upgrading the whole platform. If you're picking a version in the create-agent form and see something that isn't in the table above, your platform admin added it. The pinned `traceloop-sdk` and supported Python versions for those entries live in the operator's chart values. Operators: see [Instrumentation Catalog](/agent-manager/docs/v1.0.0-alpha1/administration/instrumentation-catalog/.md) for how to add a version, mirror images for air-gapped installs, and set the platform default.