Kubernetes

It started with a postmortem I never wanted to write. A merchant flash sale launched 20 minutes ahead of schedule. Traffic to the payment authorization services doubled in under a minute. Kubernetes HPA did exactly what it was configured to do — it detected the CPU spike and requested scale-out across over 150 checkout-path services simultaneously. Most new pods fit on existing nodes, but dozens of services exhausted their node pool headroom and triggered provisioner requests in a burst. The node provisioner stalled under the queue pressure. New capacity came up four minutes later. ...

Three ways that conventional per-service autoscaling breaks down at 100,000 pods, ~1,550 applications, and five shared domains, and why no amount of HPA tuning makes them go away. Failure Mode 1: The Provisioner Stampede HPA is designed to be autonomous. Each deployment has its own HPA object, its own target utilization, its own scale-out logic. This is great for isolation. Changes to one service’s autoscaling config do not affect others. It breaks down under coordination pressure. ...

Commander and Soldiers: Decomposing the Scaling Problem Part 3 · Series: Teaching Kubernetes to Think Ahead By Aashish Sheshadri — Platform Architecture The design space for RL-based autoscaling has three obvious options. Two of them do not hold up under scrutiny. Option 1: One Big Agent The most natural first thought: replace all 1,500 HPAs with a single RL agent that observes the entire cluster and outputs replica counts for every service. ...

The previous post ended on a claim I owe you evidence for: that getting the reward function right was the hardest part of this project. Before I can show you that, I have to show you what the Commander is actually working with — what it can see, and what it is allowed to do about it. Those two things bound everything the reward function can possibly ask for. The Commander never directly touches a replica count. It does not know how many pods service X is running. All it does is answer one question: how should I distribute the cluster’s capacity across five domains, right now, given what I know and what I expect? ...

The previous post was about writing a reward function and watching a policy exploit it. This one is about the harder question underneath: how did I know any of it was working? Not “did the loss go down.” Loss went down the entire time, including during the weeks the policy was quietly learning to game me. I mean the actual question — is this thing doing what I meant, and how would I find out if it were not. ...