“Misalignment traps lie in wait” by David Africa

Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows: Models are very sample efficient, and larger models tend to be more so. They generalise broadly from a few bits. Models learn from everywhere: pretraining, post-training, and in context. We may also figure out online learning eventually. Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent. Such settings are often considered toy, but this is an over-update. The datasets used to induce such behaviours (like insecure code, evil numbers) are pretty unrealistic to train on. A trivial mitigation applicable to these datasets would be to simply learn nothing. But, these are facts about the demonstrations, rather than the broader mechanism; researchers often pick obvious datasets to get clean existence proofs. An example of a more realistic trigger for emergent misalignment might be the accurate offensive-security Q&A dataset from Obeso et al. This is a dataset where you might plausibly want a model to learn valuable knowledge or skills, and [...] The original text contained 3 footnotes which were omitted from this narration. --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/o9CYcjZDqCQf5qeJy/misalignment-traps-lie-in-wait --- Narrated by TYPE III AUDIO.