https://riteshsonawane.com/tags/inference/Recent content in inference on Ritesh SonawaneHugoen-usMon, 06 Jul 2026 00:00:00 +0000https://riteshsonawane.com/blog/llms-inference-kpis/Mon, 06 Jul 2026 00:00:00 +0000https://riteshsonawane.com/blog/llms-inference-kpis/<p>Monitoring LLM inference is not like monitoring a conventional web service. The metrics are different, the failure modes are different, and the relationship between what you measure and what the user experiences is more indirect.</p> <p>This guide explains the most useful LLM inference KPIs for SREs, what each metric measures, what causes it to change, and how to use the signals together when production performance degrades.</p> <h2 id="the-mental-model-prefill-and-decode">The mental model: prefill and decode</h2> <p>Every LLM inference request goes through two distinct phases. If you do not understand the difference, the metrics will not make sense.</p>https://riteshsonawane.com/blog/k8s-gateway-api-inference/Fri, 26 Jun 2026 00:00:00 +0000https://riteshsonawane.com/blog/k8s-gateway-api-inference/<h2 id="kubernetes-gateway-api-inference-extension-what-it-is-and-why-it-matters">Kubernetes Gateway API Inference Extension: What It Is and Why It Matters</h2> <p>If you are running LLM workloads on Kubernetes today, you have almost certainly hit the same wall. Your standard Ingress or Gateway routes traffic to pods based on HTTP path or headers. Round-robin distributes load across replicas. Everything looks fine from the control plane perspective. But your GPU utilization is uneven, some replicas are getting hammered while others sit idle, latency is unpredictable, and you have no way to differentiate a low-latency interactive chat request from a background batch job that can afford to wait.</p>