Production-Ml

The previous post ended on a claim I owe you evidence for: that getting the reward function right was the hardest part of this project. Before I can show you that, I have to show you what the Commander is actually working with — what it can see, and what it is allowed to do about it. Those two things bound everything the reward function can possibly ask for. The Commander never directly touches a replica count. It does not know how many pods service X is running. All it does is answer one question: how should I distribute the cluster’s capacity across five domains, right now, given what I know and what I expect? ...

The previous post laid out what the Commander can see and what it is allowed to do. This one is about the part that decides what it should do, which is where the design decisions matter most and where I made nearly every mistake worth writing down. Everything below — every failure mode, every reward-hacking episode, every behavioral property — was observed in simulation during offline training. None of it happened on a production cluster. That is by construction: at a 60-second tick, the roughly five million steps this policy needed to converge would take about nine and a half years of wall-clock time to collect from the real thing. Training in production is not inadvisable here, it is arithmetically impossible. Shadow mode on production contributes under one percent of what the policy knows, and none of that percent is on-policy. ...

The previous post was about writing a reward function and watching a policy exploit it. This one is about the harder question underneath: how did I know any of it was working? Not “did the loss go down.” Loss went down the entire time, including during the weeks the policy was quietly learning to game me. I mean the actual question — is this thing doing what I meant, and how would I find out if it were not. ...