The Reward Function: Where I Made My Mistakes
The previous post laid out what the Commander can see and what it is allowed to do. This one is about the part that decides what it should do, which is where the design decisions matter most and where I made nearly every mistake worth writing down. Everything below — every failure mode, every reward-hacking episode, every behavioral property — was observed in simulation during offline training. None of it happened on a production cluster. That is by construction: at a 60-second tick, the roughly five million steps this policy needed to converge would take about nine and a half years of wall-clock time to collect from the real thing. Training in production is not inadvisable here, it is arithmetically impossible. Shadow mode on production contributes under one percent of what the policy knows, and none of that percent is on-policy. ...