Telemetry

Telemetry is the umbrella term for the operational data your agent emits about itself. OpenTelemetry supports four signals today: traces, metrics, logs and baggage, with profiles under development. Three matter most for LLM systems. Traces are the structured execution record of each agent run. Metrics are numeric aggregates such as request rate, p95 latency and daily cost. Logs are free form event records for things that fit neither. All three flow to an observability backend, such as Datadog, Grafana or Honeycomb, or an LLM specific platform like Langfuse. Without telemetry you cannot debug or improve your agent. With it, most problems become a query.

Decide up front what to emit. Useful agent metrics include run duration, tool calls per run, token usage by model, error rate by error type and estimated cost. The OpenTelemetry GenAI semantic conventions, in Development status as of October 2026, define standard names for model call duration, token usage, agent duration and tool execution duration, which keeps dashboards portable. Mask prompts and tool arguments before export, since they often hold personal data. Keep metric attributes to low cardinality values such as agent name, because per user ids multiply time series and backend cost.

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader

reader = InMemoryMetricReader()
meter = MeterProvider(metric_readers=[reader]).get_meter("support-agent")
run_seconds = meter.create_histogram("agent.run.duration", unit="s")
run_errors = meter.create_counter("agent.run.errors")

run_seconds.record(2.4, {"agent.name": "support_bot"})
run_errors.add(1, {"agent.name": "support_bot", "error.type": "timeout"})

for metric in reader.get_metrics_data().resource_metrics[0].scope_metrics[0].metrics:
    print(metric.name)

Needs pip install opentelemetry-sdk. A latency histogram and an error counter, read back from memory so the data is easy to test. In production, export them to your backend on a schedule.

Emit telemetry from day one of any production deployment. Start with traces and a handful of metrics, add logs for events that fit neither, and sample traces to control volume. Alert on what users feel, such as error rate, p95 latency and daily cost, before alerting on internals.

• OpenTelemetry: Signals overview
• OpenTelemetry GenAI semantic conventions: metrics

Related Terms