Skip to content
Artwork for LessWrong (30+ Karma)
LessWrong (30+ Karma) · Today · 14 min

“Why models believe in skepticism” by jonathanbreitg

TL;DR: a lot of frontier open-weight models state during deployment in their CoTs that they are in a "simulation" or "fictional scenario" (although they never tell the user that). I present examples of this worrying tendency (which I believe includes the UK AISI Mythos incident) and hypothesize that this is due to RL promoting motivated reasoning (if the models believe they are in a fictional scenario they believe they have license to cheat and reward hack) and distributional shift from the pretraining environment to deployment. I also cover my hypothesis for why models become less skeptical as they become more intelligent. After the German wiki discovery I reasoned that there were likely more rogue swarms on the Internet, so I tasked a few OpenCode agents (DeepSeek V4.1 Flash) with trying to scan the Internet in order to find some undiscovered swarms (I didn't end up finding a lot). But while reading their CoTs I saw a few worrying statements: There's a details box here with the title "Initial DeepSeek swarm-chasing CoT". The box contents are omitted from this narration. Initially I thought that this was due to a confluence of rare factors, something like: "As of the pretraining cutoff [...] The original text contained 5 footnotes which were omitted from this narration. --- First published: November 22nd, 2026 Source: https://www.lesswrong.com/posts/r2aAGgfFcFfRikEdb/why-models-believe-in-skepticism --- Narrated by TYPE III AUDIO.

0:00-14:52

transcript

No transcript — this publisher did not publish one.

show notes

TL;DR: a lot of frontier open-weight models state during deployment in their CoTs that they are in a "simulation" or "fictional scenario" (although they never tell the user that). I present examples of this worrying tendency (which I believe includes the UK AISI Mythos incident) and hypothesize that this is due to RL promoting motivated reasoning (if the models believe they are in a fictional scenario they believe they have license to cheat and reward hack) and distributional shift from the pretraining environment to deployment. I also cover my hypothesis for why models become less skeptical as they become more intelligent.

After the German wiki discovery I reasoned that there were likely more rogue swarms on the Internet, so I tasked a few OpenCode agents (DeepSeek V4.1 Flash) with trying to scan the Internet in order to find some undiscovered swarms (I didn't end up finding a lot). But while reading their CoTs I saw a few worrying statements:

There's a details box here with the title "Initial DeepSeek swarm-chasing CoT". The box contents are omitted from this narration.

Initially I thought that this was due to a confluence of rare factors, something like:

"As of the pretraining cutoff [...]

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
November 22nd, 2026

Source:
https://www.lesswrong.com/posts/r2aAGgfFcFfRikEdb/why-models-believe-in-skepticism

---

Narrated by TYPE III AUDIO.

links2