LLM Observability with OpenTelemetry GenAI Conventions

Offline evaluation tells you how a version performs on your test set. Observability tells you what is happening with real users right now: which requests are slow, which cost the most, where the agent looped, and which answers users disliked. This article explains the OpenTelemetry approach, what to capture, and how to add online evaluation on top. It is part of the LLM evaluation guide.

Why OpenTelemetry

OpenTelemetry is the vendor-neutral standard for traces, metrics and logs. Its generative AI semantic conventions, now maintained in a dedicated repository, define how to describe model calls, agent invocations and tool executions as spans (OpenTelemetry). Instrumenting against them avoids lock-in: the same spans can go to any backend that accepts OTLP. The conventions are still evolving, so pin the versions of the instrumentation libraries you use and check attribute names when you upgrade.

What to capture on each span

Setting it up, step by step

1. Add the OpenTelemetry SDK and an OTLP exporter to your service, with a service.name.
2. Wrap model calls, tool calls and agent steps in spans with the GenAI attributes, or use an instrumentation library that does it for you.
3. Decide on content capture. GenAI instrumentations leave message content out unless you opt in, which is the safe default.
4. Redact personal data before storage and never send API keys or bearer tokens as attributes.
5. Point the exporter at your backend with an ingest key that can write but not read.
6. Add the prompt version and conversation id as attributes so traces can be linked to changes and sessions.

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "support-bot"}))
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(
    endpoint="https://YOUR-BACKEND/v1/traces",
    headers={"x-api-key": "YOUR_WRITE_ONLY_INGEST_KEY"},
)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-bot")

with tracer.start_as_current_span("chat my-model") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.request.model", "my-model")
    # ... call the model, then record usage:
    span.set_attribute("gen_ai.usage.input_tokens", 120)
    span.set_attribute("gen_ai.usage.output_tokens", 42)

From observability to online evaluation

Traces become an evaluation signal when you score them. Online evaluation samples a share of live spans that match a rule (a service, an operation, a model), runs scorers or judge models on them, charts the average per day against a threshold, and alerts when quality drops. Keep the sample rate and a daily cap on judge calls explicit, since every judged span costs tokens. Pair it with user feedback and a human review queue for spans the judge flags, and turn confirmed failures into dataset cases.

What to alert on

• Online evaluation score below a threshold for a rule.
• Error rate, p95 latency, tokens or cost per request above a baseline window.
• A rise in negative user feedback.
• Volume anomalies, which often mean a retry loop or abuse.

LLM tracing in NexGen QA

NexGen QA accepts OpenTelemetry traces over OTLP/HTTP with a write-only ingest key, reads the GenAI attributes, and shows traces with model, service, duration, span count, tokens and estimated cost, plus cost and latency charts. Workspace admins choose whether prompts and responses are stored and can redact common personal-data patterns before storage; API keys and bearer tokens are always removed. The Python SDK sets up tracing for LangChain, LlamaIndex, the OpenAI Agents SDK, OpenAI and Anthropic with one configure call, and the server also reads OpenInference attributes. Spans that share a session id are grouped into conversations you can read turn by turn. Online evaluations (Pro) score a sampled share of live spans, or a whole conversation once it goes idle, with your checks and judge models, a daily judge-call cap and alerts when the average drops below a threshold. Drift monitors (Pro) compare score distributions, pass rates and, opt-in, prompt or response embeddings on your own key against a baseline window. Annotation queues route spans to reviewers. Free keeps 10,000 spans a month for 7 days; Pro 2,000,000 a month for 90 days. See AI evaluation in NexGen QA.