kubernetes on Ritesh Sonawane

LLM Inference KPIs Every SRE Should Know

Mon, 06 Jul 2026 00:00:00 +0000

Monitoring LLM inference is not like monitoring a conventional web service. The metrics are different, the failure modes are different, and the relationship between what you measure and what the user experiences is more indirect.

This guide explains the most useful LLM inference KPIs for SREs, what each metric measures, what causes it to change, and how to use the signals together when production performance degrades.

The mental model: prefill and decode

Every LLM inference request goes through two distinct phases. If you do not understand the difference, the metrics will not make sense.

Kubernetes Gateway API Inference Extension

Fri, 26 Jun 2026 00:00:00 +0000

Kubernetes Gateway API Inference Extension: What It Is and Why It Matters

If you are running LLM workloads on Kubernetes today, you have almost certainly hit the same wall. Your standard Ingress or Gateway routes traffic to pods based on HTTP path or headers. Round-robin distributes load across replicas. Everything looks fine from the control plane perspective. But your GPU utilization is uneven, some replicas are getting hammered while others sit idle, latency is unpredictable, and you have no way to differentiate a low-latency interactive chat request from a background batch job that can afford to wait.

ndots in Kubernetes

Tue, 07 Apr 2026 00:00:00 +0000

Let’s see how the ndots option works in Kubernetes.

In Kubernetes, we connect to running pods either directly or via a Kubernetes Service. This post focuses on how service DNS resolution actually works under the hood.

The resolv.conf file

In Linux, /etc/resolv.conf is used by the system resolver for domain name resolution. Kubernetes automatically injects this file into every pod with the configuration needed to resolve services inside the cluster.

Here is an example from a CNPG cluster pod running in the database namespace:

Streamlining Model Lifecycle with KubeRay

Fri, 24 May 2024 00:00:00 +0000

This blog is based on my work at CloudRaft!

Introduction

As the demand for ML and LLM models continues to grow, so does the need for reliability and scalability. Integrating Kubernetes into model development emerges as a powerful solution. By leveraging Kubernetes, we can streamline the process of model development, decrease costs, and enhance model reliability. This can be achieved using Ray on Kubernetes. But before that, let’s take a look at the lifecycle of an ML model.

Making Kubernetes Simple with Talos

Tue, 07 May 2024 00:00:00 +0000

This blog is based on my work at CloudRaft! (My first blog at CloudRaft)

Kubernetes marks its 10th anniversary this year with the release of version v1.30.0, solidifying its status as the cloud platform of choice. Managed Kubernetes clusters like EKS, GKS and AKS represent 73% of the total cluster, remaining 27% are self-managed as per Dynatrace. The last decade has been an era of public cloud but due to increasing costs, some businesses are trying to find a balance with the hybrid cloud. Approximately 76% of organizations now leverage multiple clouds which is a combination of public and private clouds as per VMware. Kubernetes allows us to build a multi-cloud and private cloud layer on the hardware of choice and in a cost-effective way without committing to one specific cloud.