“Gradient Routing Generalizes to Unrouted Hack Types” by jacob_drori
transcript
show notes
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand.
Introduction
Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is:
- Initialize a retain and a forget adapter.
- Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both.
- At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much.
I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with [...]
---
Outline:
(00:47) Introduction
(02:37) Methodology
(02:40) Recap
(04:00) Narrow Routing
(05:27) What I report
(07:18) Results
(10:13) Conclusion
(11:45) Appendix: Results with nudge prompt
(12:29) Appendix: Predictions
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
October 23rd, 2026
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
