Reinforcement-Learning

It started with a postmortem I never wanted to write. A merchant flash sale launched 20 minutes ahead of schedule. Traffic to the payment authorization services doubled in under a minute. Kubernetes HPA did exactly what it was configured to do — it detected the CPU spike and requested scale-out across over 150 checkout-path services simultaneously. Most new pods fit on existing nodes, but dozens of services exhausted their node pool headroom and triggered provisioner requests in a burst. The node provisioner stalled under the queue pressure. New capacity came up four minutes later. ...

Commander and Soldiers: Decomposing the Scaling Problem Part 3 · Series: Teaching Kubernetes to Think Ahead By Aashish Sheshadri — Platform Architecture The design space for RL-based autoscaling has three obvious options. Two of them do not hold up under scrutiny. Option 1: One Big Agent The most natural first thought: replace all 1,500 HPAs with a single RL agent that observes the entire cluster and outputs replica counts for every service. ...

The previous post ended on a claim I owe you evidence for: that getting the reward function right was the hardest part of this project. Before I can show you that, I have to show you what the Commander is actually working with — what it can see, and what it is allowed to do about it. Those two things bound everything the reward function can possibly ask for. The Commander never directly touches a replica count. It does not know how many pods service X is running. All it does is answer one question: how should I distribute the cluster’s capacity across five domains, right now, given what I know and what I expect? ...

The previous post laid out what the Commander can see and what it is allowed to do. This one is about the part that decides what it should do, which is where the design decisions matter most and where I made nearly every mistake worth writing down. Everything below — every failure mode, every reward-hacking episode, every behavioral property — was observed in simulation during offline training. None of it happened on a production cluster. That is by construction: at a 60-second tick, the roughly five million steps this policy needed to converge would take about nine and a half years of wall-clock time to collect from the real thing. Training in production is not inadvisable here, it is arithmetically impossible. Shadow mode on production contributes under one percent of what the policy knows, and none of that percent is on-policy. ...

The previous post was about writing a reward function and watching a policy exploit it. This one is about the harder question underneath: how did I know any of it was working? Not “did the loss go down.” Loss went down the entire time, including during the weeks the policy was quietly learning to game me. I mean the actual question — is this thing doing what I meant, and how would I find out if it were not. ...