What Changes When the Thing You Monitor Is an LLM
Notes from an observability engineer running his own inference service
I spent years in application performance monitoring at Dynatrace, working with enterprise teams on latency, service dependencies, and the gap between a dashboard that says everything is fine and a customer who has been waiting twenty seconds. Then I started running my own LLM inference service: Advisory Guide, a customer support workflow tool served with vLLM on an H100.
Some of what I knew carried over directly. Some of it pointed me at the wrong thing. This post is about the difference.
What transfers unchanged
Percentiles over averages. Measuring what the user experiences, not what the server reports. Synthetic transactions that exercise the real path. Following one request across every layer it touches. None of that changes when the workload becomes a model. If anything it matters more, because LLM requests vary in cost far more than a typical API call does.
What changes is the set of signals that tell you something is about to go wrong.
1. Latency is two numbers, and you have to pick which one you promise
A web request has one latency. A streamed LLM response has at least two: time to first token, which is when the user sees something start to happen, and full-response time, which is when the answer is usable. They fail independently. A long prompt that misses the cache hurts time to first token. A long answer hurts full-response time even when the first token arrived instantly.
For my service the answer only helps once it is complete, so I gate on full-response time, using targets taken from a recorded real call: p50 at or below 4.46 seconds, p90 at or below 7.89, and a worst case at or below 9.85. Time to first token stays on the board as a diagnostic, because it tells me whether a slowdown is coming from reading the prompt or writing the answer.
2. Utilization is not work
On a 16-GPU fine-tuning run I watched GPU utilization sit between 94 and 100 percent for 78 minutes. That number says a kernel was running. It does not say the kernel was doing useful work. The signal I trusted more was power: each H200 drew close to 670 watts against a 700-watt cap for the whole run. Busy-looking GPUs that are actually stalled show up as low power long before they show up anywhere else.
In serving, the equivalent is the scheduler. vLLM exposes how many requests are running and how many are waiting. A GPU at high utilization with a growing waiting queue is saturated. A GPU at moderate utilization with nothing waiting has room. Utilization alone cannot tell those apart.
3. The leading indicator is memory, not latency
In most services, latency is the early warning. In LLM serving it is often the last thing to move.
Every active conversation keeps a cache of its context on the GPU. When I raised the total cached context step by step, the prefix-cache hit rate held at about 83 percent right up to the size of the cache pool, then dropped to exactly zero somewhere between 1.06 and 1.20 times that size. That is classic least-recently-used behavior under a rotating workload: once the working set exceeds capacity, every request evicts the entry the next one needs. Median latency only rose by about two seconds when the cache had already collapsed.
So the alert that matters is resident context against cache capacity, measured in tokens, with a margin below the edge. By the time latency alerts fire, every user is already slower.
4. Requests are the wrong unit
Request rate is the first thing an APM engineer reaches for, and it hides the most important variable in LLM serving: how many tokens each request reads and writes.
One load test made this concrete. Two otherwise identical runs differed only in whether the production system prompt was included. Without the instructions that keep answers short, median output was 169 tokens instead of 36, longer generations held scheduler slots, and p90 full-response time went from 3.21 seconds to 23.95. Nothing about the infrastructure changed.
Two lessons came out of that. Output tokens per request belong on the dashboard next to request rate. And a prompt change is a capacity change, so it should be compared before and after a deploy the same way a code change would be.
5. Health checks have to do real work
Synthetic monitoring was one of the most valuable things I used in APM, and it matters even more here. I had a model server pass every health check while every real generation failed, because a GPU kernel compiled on first use needed a build tool that was missing from the image. In another case a new server failed to start while the old process kept answering on the same port, so the health check was green and coming from the wrong version.
A health endpoint proves the process is alive. Readiness for an inference service means a representative request completes and the responder reports which release it is.
6. The dashboards I would build
For an executive, three numbers: full-response p95 against the target, the share of requests that complete successfully, and cost per active user. Those describe whether customers are being served and what it costs, without needing to know what a KV cache is.
For the engineer on call, three more: resident tokens as a fraction of the cache pool, waiting versus running requests, and output tokens per request by release. Those explain why the first three moved, and the first one usually moves first.
What surprised me most
In traditional services, degradation is usually gradual. Queues lengthen, latency creeps up, and you have time to react. The most important failure in my inference service did not work that way. The cache held, and then it was gone. The single biggest change in how I monitor is treating capacity as a wall to stay behind, not a slope to watch.
The measurements behind this post are in my write-up on holding long conversations in GPU memory and the 16-GPU fine-tuning run.