Understanding LLM Inference KPIs: From TTFT to KV Cache
Monitoring LLM inference is not like monitoring a web service. The metrics are different, the failure modes are different, and the relationship between what you measure and what the user experiences is more indirect. This post covers the core KPIs, what they actually measure, and how to reason about them operationally.
First, Understand the Two Phases
Every LLM inference request goes through two distinct phases. If you do not understand the difference, the metrics will not make sense.
Prefill takes the input prompt, runs it through the model, and builds the KV cache. It processes all input tokens in parallel. This phase is compute-bound: the GPU is doing a large matrix multiplication across all input tokens simultaneously. A 100-token prompt results in 100 K/V vector pairs stored in the KV cache.
Decode generates the output one token at a time. Each new token looks at the full KV cache built during prefill, then appends its own K/V vectors for the next token to use. This phase is memory-bandwidth-bound: the GPU spends most of its time reading the KV cache from HBM (high-bandwidth memory) rather than doing heavy compute.
User Prompt
│
▼
Prefill
(Read all input tokens + Build KV cache)
│
▼
First output token ← TTFT ends here
│
▼
Decode
(Token 2 → Token 3 → Token 4 ...)
│
▼
Time between tokens = TPOT
This distinction matters because prefill and decode are sensitive to completely different hardware characteristics and are improved by completely different interventions. Almost every KPI maps to one of these two phases.
The Core KPIs
TTFT (Time To First Token)
What it measures: Time from when the client sends the request until the first output token is received.
TTFT covers the entire prefill phase plus any time the request spent waiting in a queue before prefill began. The model cannot generate the first token until it has processed the entire input prompt and built the KV cache, so TTFT is directly proportional to prompt length.
Why it matters: This is what the user feels as “how long did I wait before anything happened.” For interactive use cases, high TTFT makes the product feel broken even if token generation is fast afterward.
What drives it up:
- Long input prompts (more compute needed in prefill)
- High queue depth (request waited before getting a GPU)
- Underpowered prefill compute relative to prompt length
What to alert on: p50 and p99 TTFT separately. p99 TTFT that is significantly higher than p50 usually means queue spikes under load, not a prefill performance issue.
Prefill Latency
What it measures: Time to process the input prompt and build the KV cache, isolated from queue time.
Prefill latency is the compute portion of TTFT. The difference between TTFT and prefill latency is queue time: how long the request waited before prefill started.
Why it matters: If TTFT is high, you need to know whether it is because prefill is slow (hardware or model issue) or because the request waited in a queue (capacity issue). These require different interventions.
What drives it up:
- Very long prompts
- Insufficient compute on the serving node
- Model served without tensor parallelism when it needs it
TPOT (Time Per Output Token)
What it measures: Average time to generate each output token after the first one, during the decode phase.
TPOT is the per-token decode speed. It reflects how fast the GPU can read the KV cache from HBM and generate the next token. Since decode is memory-bandwidth-bound, TPOT is more sensitive to memory bandwidth utilization and concurrent request count than to raw compute.
Why it matters: TPOT determines the streaming speed the user sees. A TPOT of 50ms means the user sees roughly 20 tokens per second, which feels smooth. A TPOT of 200ms means 5 tokens per second, which feels sluggish.
What drives it up:
- High concurrent request count (KV cache for many sessions must be read together)
- Large KV cache size (more data to read per token)
- Memory bandwidth saturation on the GPU
- Large batch sizes that cause inter-request interference during decode
What to alert on: p50 and p99 TPOT. Unlike TTFT, TPOT degradation is usually gradual as load increases rather than spiky. A slow upward trend in TPOT p50 over hours typically indicates growing concurrent session count.
Throughput (TPS and RPS)
TPS (Tokens Per Second): Total output tokens generated per second across all requests. This is the headline capacity metric for your serving cluster.
RPS/QPS (Requests Per Second): Number of complete requests handled per second. Less useful than TPS for LLM serving because requests vary enormously in output length, so two RPS numbers from different workloads are not comparable.
Why throughput matters: TPS tells you how much useful work your GPU fleet is doing. It is the denominator in your cost-per-token calculation. Optimizing TPS directly reduces infrastructure cost per unit of output.
What drives TPS down:
- Low GPU utilization (GPUs idle waiting for requests)
- Poor batching (requests not being grouped efficiently)
- Memory bandwidth saturation causing TPOT degradation
- High queue time reducing time GPUs spend actually generating
GPU Utilization
What it measures: Percentage of time the GPU’s compute units are actively processing work.
GPU utilization is the efficiency signal for your fleet. A GPU at 30% utilization is being wasted. A GPU at 95% is healthy but getting close to a throughput ceiling.
What to watch for: GPU utilization alone is not sufficient. A GPU can show 95% utilization but still be slow if it is memory-bandwidth-bound during decode rather than compute-bound during prefill. You need to correlate GPU utilization with TPOT to understand whether the GPU is actually doing useful work or waiting on memory reads.
What drives it down:
- Insufficient request volume (underpowered fleet or traffic dropped)
- Queue time due to scheduler inefficiency
- Cold-start periods after replica restarts
HBM Memory Usage
What it measures: Total GPU memory consumed by model weights, KV cache, and activations.
HBM (High-Bandwidth Memory) is the VRAM on the GPU. It holds three things:
- Model weights: Fixed cost. A 70B parameter model in 16-bit precision needs roughly 140GB. This never changes.
- KV cache: Dynamic cost. Grows with the number of active sessions and their context lengths.
- Activations: Per-request transient memory used during forward pass computation.
Why it matters: HBM is the binding constraint for LLM serving. When you run out of HBM, the inference server either evicts KV cache entries (causing recomputation and latency spikes) or the process OOMs and crashes.
What to alert on: HBM utilization above 80%. The gap between 80% and 100% can close very quickly under a burst of long-context requests. Alert at 80% to give yourself time to react.
HBM Bandwidth Utilization
What it measures: How much of the GPU memory bandwidth is being used, expressed as a percentage of theoretical maximum.
This is distinct from HBM memory usage. Memory usage is how much space is occupied. Memory bandwidth utilization is how fast data is being read from and written to that memory.
During decode, each token generation requires reading the entire KV cache for all active sessions from HBM. If you have 50 concurrent sessions each with 4K tokens of context, the GPU reads a significant amount of data from HBM on every decode step. When HBM bandwidth saturates, TPOT increases even if HBM memory usage and GPU compute utilization look healthy.
Why it matters for SREs: HBM bandwidth saturation is a common and easily missed cause of TPOT degradation. If your TPOT is increasing and GPU utilization looks normal, check HBM bandwidth utilization. This is the metric that makes the memory-bandwidth-bound nature of decode visible.
KV Cache Hit Rate
What it measures: Percentage of requests that reuse an existing KV cache entry for a matching prompt prefix, rather than recomputing it from scratch.
When prompt caching is enabled, the inference server stores KV cache entries for processed prefixes. If a new request shares a prefix with a cached entry (for example, a long system prompt that many requests share), the server skips prefill for the cached portion and reads the KV vectors directly. This reduces TTFT and frees up compute for other requests.
Why it matters: A high KV cache hit rate is a force multiplier on prefill capacity. If 80% of your requests share a common system prompt, a high hit rate means you only pay the compute cost of that system prompt once rather than on every request.
What drives it down:
- Replica restarts clearing in-memory caches
- Traffic spreading across replicas without prefix-aware routing (a request with a cached prefix landing on a replica that does not have the cache)
- Diverse prompts with no common prefix
Operational note: If you see KV cache hit rate drop suddenly without any obvious traffic change, check for recent replica restarts. A rolling restart of your inference deployment will cause a temporary hit rate collapse as all in-memory caches are cleared.
KV Cache Size
What it measures: Total GPU memory occupied by the KV cache across all active sessions.
KV cache size grows with both the number of active sessions and the average context length of those sessions. A workload with long context windows (32K tokens or more) can consume vastly more KV cache memory than a workload with short prompts, even at the same request rate.
Why it matters: KV cache competes directly with model weights for HBM. You cannot reduce model weight memory without changing the model, but KV cache size is driven by workload characteristics that can change suddenly. If users start sending longer prompts, KV cache size can grow substantially without any change in request volume.
What to monitor: Track KV cache size as a percentage of total HBM rather than as an absolute number. When KV cache plus model weights approaches 90% of total HBM, you are close to the OOM boundary.
Batch Size
What it measures: Number of requests being processed together in a single inference forward pass.
Modern inference servers use continuous batching, which groups multiple in-flight requests into a single GPU operation during both prefill and decode. Larger batches improve GPU utilization and TPS but also increase per-request memory consumption and can introduce head-of-line blocking where short requests wait behind long ones.
What to watch: If batch size is consistently at the configured maximum, you are either compute-saturated or memory-saturated. Distinguish the two by checking whether TPOT or TTFT is degrading first.
Queue Time
What it measures: Time a request waits before a GPU slot becomes available to start processing it.
Queue time is the component of TTFT that is not prefill latency. A request arriving at a fully loaded server must wait until an existing request completes and frees capacity.
Why it matters: Queue time is the first signal of capacity exhaustion. It rises before GPU utilization hits 100% because requests queue even when GPUs are at 90% utilization if new arrivals cannot be batched into current work.
What to alert on: Any sustained non-zero queue time at p50 is a capacity signal. Spiky p99 queue time with low p50 usually means bursty traffic exceeding burst capacity. Consistently high p50 queue time means you are consistently undersized.
Concurrent Requests
What it measures: Number of requests actively being served at the same time, including both prefill and decode phases.
This is distinct from RPS. A 30-second request contributes 30 seconds of concurrency regardless of when it arrived. High concurrency directly drives up KV cache size and HBM bandwidth utilization.
Operational use: Concurrent requests is the leading indicator for memory pressure. When it rises, KV cache grows, HBM bandwidth utilization increases, and TPOT degrades. Tracking concurrent requests lets you predict TPOT degradation before it shows up in latency metrics.
How the KPIs Relate to Each Other
These metrics do not exist in isolation. Understanding the causal chain helps you diagnose problems faster.
Load increases (more requests arrive):
- Concurrent requests increase
- KV cache size grows
- HBM bandwidth utilization rises
- TPOT increases
- Queue time increases
- TTFT increases (queue time component)
Long prompts arrive:
- Prefill latency increases
- TTFT increases
- KV cache size grows per request
- HBM memory usage increases
Replica restart:
- KV cache hit rate drops
- TTFT increases temporarily (prefill runs for every request that previously hit cache)
Memory pressure approaches limit:
- KV cache evictions begin
- Recomputation of evicted entries causes TTFT spikes
- If not caught, OOM kills follow
Minimum Viable Dashboard
If you are setting up monitoring for an LLM serving stack, these are the metrics to instrument first, in priority order:
- TTFT p50 and p99
- TPOT p50 and p99
- HBM memory usage per GPU
- HBM bandwidth utilization per GPU
- KV cache utilization (as percentage of HBM)
- Request queue depth
- TPS (output tokens per second)
- KV cache hit rate
- Concurrent active requests
- Finish reason distribution (completed, truncated, error)
These ten give you enough signal to understand whether the system is healthy, where bottlenecks are forming, and what class of problem you are dealing with when something degrades.