1.
ROUTE PRODUCTION TRAFFIC
Shift-1 serves every request, with a little exploration
2.
LABEL REAL TURNS
Developer feedback and hindsight judgments
3.
RETRAIN THE ROUTER
Fit success predictors on the fresh labels
4.
DEPLOY THE UPDATE
The improved router takes over routing
BRANCHED CONTINUATIONS
Costly, but the cleanest Q labels; they anchor the cheaper ones
EVALUATE THE CURRENT ROUTER
IMPROVE ON IT
CALIBRATE
APPROXIMATE POLICY ITERATION, RUN ONLINE
Each pass grounds the next router in how the current one actually performed.