AI inference: observability for operations

AWS is extending SageMaker AI with detailed metrics for generative inference and a SageMaker insights dashboard in Amazon CloudWatch. This is not a cosmetic monitoring feature. It shows where operating LLM applications is heading: away from “the API responds or it doesn’t” toward measurable inference quality, capacity and reliability.

What AWS adds

According to AWS, SageMaker inference endpoints now emit native OpenTelemetry metrics to CloudWatch. The new dashboard organizes them on three levels: performance, capacity and reliability. Visible signals include token latencies, throughput, errors, GPU, CPU and memory utilization, KV cache pressure, availability zone distribution, scaling events and cold-start diagnostics.

The classic CloudWatch documentation remains important here: metrics such as ModelLatency and OverheadLatency still describe where time is lost. What is new is above all the depth for generative workloads, where token streaming, GPU memory and inference components determine user experience and cost.

Why this is more than AWS

For Azure AI Foundry, Microsoft likewise describes tracing via OpenTelemetry and Application Insights: timeline, spans, inputs, outputs, errors, latencies and token usage become visible during development and operations. In parallel, OpenTelemetry maintains its own semantic conventions for Generative AI, Agents, MCP, Anthropic, AWS Bedrock, Azure AI Inference and OpenAI.

The common denominator: LLM operations become observable like classic software — but with new signals. A high response time can stem from the model, the prompt, a tool call, GPU capacity, the cache or a poorly planned agent loop.

What DACH companies should review

For regulated organizations, a chatbot dashboard is not enough. They need SLOs for response time and error rate, cost metrics per business unit, data protection rules for trace content and clear retention requirements. Prompt and response content in particular must not end up uncontrolled in telemetry.

Start with a small mandatory set: token consumption, P95/P99 latency, error classes, tool calls, cache hits, cost per operation and an alert when an agent runs an unusually high number of steps. The operational question is no longer “Which model do we use?” but “Can we explain what our AI system did when something went wrong?”