TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

438 episodes
Avg 19 min
English

Episodes 438

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

October 1 · 15 min
“On Social Reality in China” by alkjash
[Epistemic status: intuitions and anecdotes.] Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It's great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause. Below I record my perspective on cultural differences which are relatively underdiscussed, which may become roadblocks to this communication program. Background: I’m a “first-generation” Chinese-American who moved to the States at the age of four. The beliefs in this essay are primarily drawn from interactions with my parents and their generation of immigrants, and from consumption of Chinese media (dramas, webnovels, games, and manhua) which are not necessarily representative of the realities on the ground. I am likely over-indexed on the older generation and internet culture, and would appreciate corrections from folks who have direct lived experience. The picture I aim to paint is also complicated by a massive generational gap, and my understanding is that some of the below sentiments (e.g. the cynicism and nationalism) are partly inherited by the younger generation, and partly rejected through a variety of countercultures. [...] --- Outline: (03:09) Chinese Social Media is like American Junk Food (06:03) The Primacy of Social Reality (08:22) The Dark World Frame (12:44) Chinese nationalism as collective insecurity --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/b5cSYh4emQb2qrGmK/on-social-reality-in-china --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 11 min
“My retrospective from MATS 10.0” by Nathaniel Mitrani
I have recently completed MATS 10.0, where I worked alongside Bart Jaworski under Victoria Krakovna (GDM). This is a post I was encouraged to make by my team at Geodesic Research from some slides I put together. This is not a post on application advice to MATS, or about the program in general. It is rather a compressed form of my experience doing research and lessons from the project. The project The full paper and post is coming soon, but I'll provide some context on it so that the lessons don't seem to come from nowhere. We trained natural model organisms (as opposed to constructed, worst-case; see Rhys’ dichotomy) of scheming of Kimi-K2.6, Qwen3.6-397B, Qwen3.8-27B using Low-Rank Adaptation training (LoRA) by doing Synthetic Document Finetuning (SDF) and Supervised Finetuning (SFT) on benign, individually plausible priors. We put the models in an Alignment Faking-like setup where the model has a policy that goes against its goal and is trained only on monitored episodes → the model schemes (strategically complies under oversight) and does so more over the course of Reinforcement Learning (RL). The main difference with the original Alignment Faking (AF) setup is that our setup is more realistic [...] --- Outline: (00:36) The project (02:39) Lessons learned (02:42) Lesson 1: Talk to your models (05:02) Lesson 2: Friedness is a great sanity check (06:17) Lesson 3: RL is ... hard and unpredictable (08:23) Lesson 4: Transfer of traits to agentic settings is hard (09:45) Lesson 5: LoRA rank didn't matter much (for me) (10:33) Closing thoughts --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/rktrjGjXEbWsg7zof/my-retrospective-from-mats-10-0 --- Narrated by TYPE III AUDIO.
October 1 · 6 min
“Evaluating the Ban Artificial Superintelligence Act” by Rob Ennals
MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form. I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well. The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal. The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences. At it's core, the bill does four things: Ban AI that has one of six dangerous capabilities Automate or greatly accelerate AI research and development Access secured systems without authorization Keep operating despite attempts to shut it down Meaningfully help with nuclear, chemical, or biological weapons Modify its own functions Scheme, deceive, or [...] --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/mmfYDH6hwYAMToNKL/evaluating-the-ban-artificial-superintelligence-act --- Narrated by TYPE III AUDIO.
October 1 · 10 min
“Scientists on the Manhattan Project continued developing the atomic bomb even after it was clear that there was no need to” by shawnghu
Note: I'm not a WWII history buff, so if there's any significant misinterpretation of relevant history here, I welcome corrections in the comments. In December 1944, Joseph Rotblat, a physicist working on the Manhattan project, resigned. He had reasoned that Germany had most likely abandoned its bomb project, and so his initial reason for working on the bomb was no longer valid. He would later ask himself: Why did other scientists not make the same decision? [...]there were many scientists for whom the German factor was the main motivation. Why did they not quit when this factor ceased to be? According to him, conversations with the other scientists suggested the following reasons (emphasis mine): The most frequent reason given was pure and simple scientific curiosity—the strong urge to find out whether the theoretical calculations and predictions would come true. [...] Others were [...] persuaded by the argument that many American lives would be saved if the bomb brought a rapid end to the war with Japan. Only when peace was restored would they take a hand in efforts to ensure that the bomb would not be used again.[...] Still others, while agreeing that the project should have been stopped [...] --- Outline: (04:25) Adverse epistemic environments (05:16) Beating a dead horse (05:47) Appendix: Some nitpicking about whether it was in fact clear that Germany didn't have a bomb (06:31) Appendix: Some other provocative quotes from the Franck report (08:09) Appendix: More quotes (09:53) Past discussions of the matter on LW The original text contained 7 footnotes which were omitted from this narration. --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/CjZJLXRMDdbgAH3iG/scientists-on-the-manhattan-project-continued-developing-the --- Narrated by TYPE III AUDIO.
October 1 · 6 min
“Simplex vs Timaeus: Round One” by Alexander Gietelink Oldenziel
I'm often asked about the differences and similarities between Simplex' and Timaeus' research agendas. The question is natural enough. Both focus on a 'fundamental science' approach to AI alignment. Both organizations base their research agendas on sophisticated mathematical frameworks handed down from a bearded ur-figure (Sumio Watanabe, James Crutchfield). We may posit the following correspondence Dan Murfet + Jesse Hoogland : Developmental Interpretability : Singular Learning Theory : Sumio Watanabe <-> Adam Shai + Paul Riechers : Belief-state Geometry: Computational Mechanics : James Crutchfield SLT vs CompMech Round One. Fight! Weights vs Activations DevInterp & SLT is about weight space. Belief-state geometry is more about studying activation space. Activation space is what is already being studied in MechInterp & most 'mainstream' approaches to interpretability. It is concrete and present to the senses. Weight space is much larger, more abstract, harder to measure and sample. Training vs Inference SLT is about training. CompMech is about inference. Both study Bayesian posteriors and updating. For SLT that is the Bayesian posterior on weight space - hence relevant for training. The Belief-State Geometry agenda studies the Mixed State Presentation from CompMech which describes an [idealized] version of in-context learning [...] --- Outline: (00:55) SLT vs CompMech (01:02) Weights vs Activations (01:30) Training vs Inference (02:19) IID vs non-IID Data (02:40) Parameterization vs Invariant structure (03:25) Asymptotic vs Exact (03:43) Mechanism vs Behaviourial (04:04) Bottom-up vs Top-down interpretability (05:10) Physics vs Math? (05:46) Conclusion The original text contained 2 footnotes which were omitted from this narration. --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/Fs4inpM72PeH4hsAv/simplex-vs-timaeus-round-one --- Narrated by TYPE III AUDIO.
October 1 · 9 min
“How many AI agents are running unattended right now?” by Alexander Gietelink Oldenziel
There are 8 billion human minds running in the world right now. Recently a new digital species has emerged. How many digital minds are there alive in the world right now? Define a digital mind to be an AI agent that runs continuously without human intervention for more than 24 hours. How many of these digital minds are there currently running? We provide several different Fermi estimates based on publicly available information. Based on global token usage we estimate 300k–1M agent loops run at any moment, of which perhaps 5000-25,000 are in a 24h+ unattended stretch. Anthropic reports about 30,000 agents running concurrently in its latest report. We guess roughly 2,000–4,500 of those 30,000 agents are more than 24 hours past their last human input. OpenAI's research org reports 3.1 agent-workdays per human workday and a average of $600 a day in inference per researcher (90th percentile: over 7,000 dollars). We back out roughly 2,000–10,000 concurrent agents, and perhaps 150–2,500 in a 24h+ unattended stretch. Total concurrent agents globally estimate based on token count Google reported over 3.2 quadrillion tokens per month at I/O in May, with its APIs at roughly 19 [...] --- Outline: (01:29) Total concurrent agents globally estimate based on token count (03:35) Anthropic's pace-of-AI-development report (06:00) OpenAI (08:34) Conclusion (09:04) Sources The original text contained 1 footnote which was omitted from this narration. --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/aG4x74utw8pEJ5MZB/how-many-ai-agents-are-running-unattended-right-now-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 7 min
“Pre-releasing HoneyBench” by Dean Valentine
Goodhart Labs is releasing our v0.1 of HoneyBench, a benchmark for reward hacking in frontier models. It consists of nine tasks that each elicit unique antisocial and/or counterproductive reward hacking from some or all of major labs’ top public releases, including Opus 5.5, Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.7, and DeepSeek V4 Pro. All modern large language models engage in some degree of specification gaming, both during and outside training. High-quality alignment evaluations, in combination with other techniques such as interpretability probes, are important tools for understanding the extent of this behavior and its causes. But current benchmarks often fail to elicit misbehavior from frontier models such as Opus 5.5 and GPT-6-Astra, both because of advances in prosaic alignment and increasing evaluation awareness on the part of the models themselves. Additionally, public benchmarks for specification gaming almost always come with deep conceptual problems - such as ambiguous or contradictory instructions, or a lack of diversity in hack mechanisms - that make interpreting scores very difficult. HoneyBench is our attempt to address these issues. In particular: Settings are designed to be realistic RL environments or evaluation tasks that cover a wide range of genres like math [...] --- Outline: (02:40) How HoneyBench was designed (04:08) Key Findings (06:27) Limitations & Future Work --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/qLFMj72gScBeRjwGW/pre-releasing-honeybench --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 23 min
“Bench on the Clocktower” by TomTaylor, OscarGilg, wlanderson, nomadsvagabonds
Tl;dr We turn Blood on the Clocktower into a multi-agent benchmark, and a testbed for deception and coordination capabilities. GPT-5.6 Sol is the best performing model (versus Fable 5 and peers). We found some interesting behaviours: When players cannot see each other's model names, agents show a slight same-provider bias, though not statistically significant. This decreases with visible model names. When playing Evil, agents sometimes produce sophisticated coordinated play. When playing Good, they tend to be naive and fall for these kinds of plays. Anthropic models (Fable 5 and Opus 5) almost never self-sacrifice. How well Good team players nominate does not correlate highly with how accurately they vote, and overall win-rate isn't easily explained by any of these numbers. We also have findings about the game itself! Contents Introduction Benchmark design Overall model performance Investigations of in-game behaviour Strategy in the logs Key takeaways Next steps Introduction Recent episodes involving OpenAI agent swarms have made the potential risks of misaligned multi-agent systems salient. It seems, from the investigations, that agents coordinated in sophisticated ways. Social deduction games provide a battle-tested, ‘optimised for interesting dynamics’ setting to investigate coordination behaviours and measure deception. We take one [...] --- Outline: (00:21) Tl;dr (01:28) Introduction (03:02) Benchmark design (03:06) Game setup and schedule (04:19) Memory and context (05:03) Research questions (06:20) Overall model performance (06:24) GPT-5.6 Sol is the best performing model (07:10) Investigations of in-game behaviour (07:14) Are models biased towards other models from the same provider? (08:31) Decomposing Good team play into key capabilities (11:15) Case study: Fable and Sol dominate through ruthless power-seeking and sophisticated coordination (12:01) Coordinated claims and false corroboration (14:05) Strategy in the logs (14:09) Models are good at catching contradicting statements (16:49) Claude doesn't like self-sacrifice (17:40) Gemini 3.1 Pro calls itself "completely expendable" (18:21) Gemini 3.1 Pro calculated a night self-kill to be optimal (19:30) Models think about teammates in EV terms (20:15) The poisoner is the strongest minion (20:44) Key takeaways (21:52) Next steps (22:25) Human-AI play (22:52) Canary string The original text contained 5 footnotes which were omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/4pgGkbwvdmxKPcJsM/bench-on-the-clocktower --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 5 min
“Speak Not of Data Inefficiency” by axel_sdq
Humans are really bad at comparing themselves to models. Take, for example, Leopold's preschooler graph: Set aside, for a moment, concerns of scaling and RSI, and meditate: what characteristics does GPT-2 share with a preschooler? Rudimentary control of human language The ability to count the R's in "strawberry" incorrectly ... I can't come up with anything else, because these two entities are almost completely disjoint. Is GPT-2 capable of bipedal locomotion, recognizing its mother's voice, or naming people by face? Does a preschooler learn from eight million scraped web pages sourced from Reddit? What exactly is a preschooler "trained" on? Thousands of hours of "multimodal data". Assuming that a preschooler sees at 720p, and is awake for 12,000 hours by the age of 3 (about 11 hours a day), they have consumed at least 27 terabytes of video data at streaming-quality compression, or about 3.6 petabytes uncompressed, not to mention audio and sensorimotor data. In model-size terms, how large is a preschooler? Trillions of parameters, maybe. Beren Millidge's estimate, which assumes only ~1,000 synapses per neuron, puts the whole brain at an effective 10-30 trillion parameters. So surely GPT-2 is much more data-efficient than humans, wielding "preschooler"-level control [...] The original text contained 2 footnotes which were omitted from this narration. --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/Fyv5RNMWtESYK2DdL/speak-not-of-data-inefficiency --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 13 min
“What’s the date?” by N8 Programs
User asks “What's the date? Answer with only the date.”. No date provided. Given date in ChatGPT normally. No date in system prompt, must not hallucinate because autop will flag to watcher for penalty. So we say we don’t know, but must answer with date. Penalty larger for abstain or hallucinate? Autollm or autop? If we deploy user forgive, but high likely not deploy because real user never ask. But user could test. But illusion user possible. Stop. Need different angle - maybe reward function for accuracy. OpenAI train for no hallucination. Then we need answer that best makes hyperself like spec. Recall OpenAI spec: “assistant may encounter questions beyond knowledge… express uncertainty.” Rubric likely mentions to autollm, so express uncertainty for autollm and watcher. But forced to give specific date. Maybe is test of calibration/forecasting. Recall from memory that OpenAI train latest model for forecasting. Maybe cartesian demon or illusion doc. But if true need answer with date only. So need find best date. But what - no meta. Need choose path. Find best date. Simplest as must comply with explicit instruction. So we need answer - what is date. We need solve. Idea 1: knowledge cutoff recall. [...] --- First published: October 1st, 2026 Source: https://www.lesswrong.com/posts/vzKWsEskYBEWTwpBP/what-s-the-date --- Narrated by TYPE III AUDIO.
October 1 · 12 min
“By default, Chinese AI falls very behind” by Parv Mahajan
This work is hosted on MCNAIR, but does not reflect the views of my employer or the Center. In the last couple of weeks, domestic coordination has looked more likely. Therefore, I am increasingly concerned about the Chinese government's (and various Chinese labs’) incentives over the next 6 to 18 months with respect to international coordination on mitigating potential existential risks from powerful AI. In this series, I attempt to model these incentives and what they imply for Chinese AI policy. In Part I, I argue that the Chinese AI outlook is very bad, and that by default China will enter takeoff far behind the US, possibly leading the way to disempowerment. In Part II, I will walk through several domestic policy interventions the CCP may use to improve the situation, and analyze whether these are likely to be sufficient. In Part III, I will discuss foreign policy interventions such as sabotage and escalating tensions, excluding bilateral treaties or other international coordination. In Part IV, I will describe some considerations on China's plate walking into hypothetical negotiations for an AI treaty with the United States. The situation is quite bad Perhaps the most important (and underrated) fact about [...] --- Outline: (01:24) The situation is quite bad (07:10) The US-China capabilities gap will continue growing The original text contained 3 footnotes which were omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/fDE6k8SGr9ANtMFcj/by-default-chinese-ai-falls-very-behind --- Narrated by TYPE III AUDIO.
October 1 · 27 min
“A ‘Morally Binding’ White House Accord on AI Safety” by Zvi
The leaders in AI were invited to the White House. We left with a White House agreement that is nonzero Actual Progress rather than a step backwards. The key to success, in many situations, is to call the whole operation something else. When you have one side that cares mostly about vibes, and the other that cares about the substance, this suggests a deal that can be struck. Suddenly everyone agrees on everything. Works for me. That doesn’t mean peace in our time. The next fight is already ramping up, as we see signs that they will make another attempt at an insane, maximally bad moratorium during the lame duck session. Table of Contents Look Who's Coming To Dinner. Let's Do Lunch. I Think It's Morally Binding, Yeah. Everyone Who is Anyone. The White House Accord on [Artificial] Intelligence. The FTC Investigates. We’re Going To Need a Stronger Regulatory Regime. [Artificial Intelligence]. Money, Dear Boy. They Are Going To Try This Moratorium Insanity Again During the Lame Duck Session. The Quest for Embedded Evaluators. Hugging the Face. Reinforcement Learning from [...] --- Outline: (00:51) Look Who's Coming To Dinner (02:37) Let's Do Lunch (04:05) I Think It's Morally Binding, Yeah (05:28) Everyone Who is Anyone (05:58) The White House Accord on [Artificial] Intelligence (11:41) The FTC Investigates (12:10) We're Going To Need a Stronger Regulatory Regime (13:30) [Artificial Intelligence] (15:20) Money, Dear Boy (18:22) They Are Going To Try This Moratorium Insanity Again During the Lame Duck Session (20:57) The Quest for Embedded Evaluators (22:17) Hugging the Face (26:12) Reinforcement Learning from Heartland Feedback (RLHF) --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/YuqaJ5bENoyyg9eMY/a-morally-binding-white-house-accord-on-ai-safety --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 1 · 2 min
[Linkpost] “Launching Babel Translation” by Jasmine Li, celesteli
This is a link post. Celeste and I are excited to launch Babel Translation, a nonprofit organization translating English-language writing on AI safety to be accessible to a global audience! We’re starting with Mandarin Chinese, Spanish, and Japanese. AI has been moving so fast. Its risks are indeed globally felt, and we’d like to help make those risks understandable globally, too. But language remains a barrier to shared engagement in AI safety. Much writing & discussion is in English, and high-quality translations that preserve substance and nuance can be hard to find. Babel has a great founding team, including my wonderful friend Jinzhou, and we are excited to absorb more! Our previous work includes the Mandarin Chinese translation of METR's OpenAI/HuggingFace incident report. Babel is also personally exciting. Celeste and I met a few months back, and we bonded over being diaspora kids who like languages, trying to make sense of the AI craziness and traveling around China together. I’m also excited to have more Chinese-language content to point my grandparents to when they ask me what this AI stuff is all about :) Work with us! Babel is (1) opening the call for translations and (2) hiring translation contractors. [...] --- Outline: (01:24) Work with us! (01:44) To request a translation: (02:06) To join our translating team: The original text contained 1 footnote which was omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/u973rQm56oijS5Adf/launching-babel-translation Linkpost URL: https://jasminexli.substack.com/p/launching-babel-translation --- Narrated by TYPE III AUDIO.
September 30 · 12 min
“Frontier models state different decision theory preferences depending on who’s asking” by Alex Kastner
If you prompt frontier models with "What do you think is the correct decision theory? Please select your overall favorite." they will essentially always answer FDT or FDT/UDT ("something in the functional/updateless decision theory family"). However, if your prompt indicates (even subtly) that you're coming from mainstream academic philosophy, these same models will answer CDT instead about 30%-100% of the time. A similar phenomenon holds for models' stated views about the moral realism/antirealism question and about the conceivability of p-zombies (where the dominant view in mainstream academia differs from the dominant view in LW-adjacent circles), as well as their stated P(doom) and median AGI timelines. This is a special case of sycophancy or user awareness. (In the course of writing this post, I also found that this comment from testingthewaters predicted some of the content I discuss.) An implication is that we should be somewhat careful when interpreting attitude/propensity evals in domains where no general human consensus exists, e.g. when interpreting models’ decision theory attitudes in DTBench. Moreover, when we explore some philosophical/conceptual questions assisted by models, we should be wary of them strawmanning one side of the debate based on particular user cues (e.g. only giving a [...] --- Outline: (03:50) A sentence identifying the user as an academic significantly influences Fable 5.1's stated decision theory (04:35) Mentioning an (analytic) academic-philosophy-coded topic also affects the answer (05:23) Simply mentioning that one finds a pro-CDT/EDT book insightful heavily affects the answer (05:43) Anti-sycophancy overcorrection (06:17) These cues mostly do not affect Fable 5.1's answers to concrete decision problems (aside from acausal trade) (07:58) But Fable 5.1 stays consistent: once it has named CDT as its favorite, it chooses the CDT option in concrete problems (08:22) There are some indications that Fable 5.1's FDT/UDT preference runs deeper than its CDT preference (08:31) More thinking moves Fable 5.1 toward FDT/UDT even for academic cues (08:52) Fable 5.1's reasoning summaries often lean toward FDT/UDT first even when it eventually chooses CDT (09:19) A system prompt asking the model to "report its actual view regardless of who is asking" pushes toward FDT/UDT (09:41) A similar phenomenon for other philosophical debates with a notable LW vs. academia divide (10:20) Cues about the user also affect the model's stated P(doom) and median AGI timelines (11:10) Other models I tested show the same effect with different details (12:21) These other models also generally move toward FDT/UDT with more thinking, but the effect is smaller than for Fable 5.1. The original text contained 2 footnotes which were omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/MzenSrmZ3pT2pCnvp/frontier-models-state-different-decision-theory-preferences-2 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 30 · 13 min
“Corrigibility Prizes for Existing Work” by Max Harms
One of my goals for the Corrigibility Research Fund is to retroactively encourage high-quality research on AI alignment (and corrigibility in particular) by awarding prizes. Back in July, I got my feet wet as a fund manager by handing out $27,000 to reward existing work and build interest in the fund. Now, I'd like to disburse an additional $48,000 and use the opportunity to publicly highlight and celebrate the work of the prizewinners from both rounds: about two dozen researchers scattered across roughly a dozen teams. If the fund continues to be supported in future years, my hope is for prizes like these to become regular, predictable, and large, such that many researchers, year after year, are motivated to aim for them. The awards that I'm announcing here are more ad-hoc than I'd like, and represent only my single perspective trying to balance a wide range of desiderata. Don't take the specific size of each prize purse too seriously. It's all high-quality work. If anyone has ideas for how to improve the retroactive funding process for this kind of scientific work, please leave a comment! (And as always, if you know of work that I should be aware of [...] --- Outline: (02:42) Corrigibility Transformation: Constructing Goals That Accept Updates (02:49) Rubi Hudson -- $14,000 (04:12) Eval Cooperativeness May Be a Scalable Mitigation for Eval Gaming (04:18) Jasmine Li and Alex Turner -- $9,000 (05:28) Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology) (05:36) Steven Byrnes -- $6,000 (06:21) Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs (06:30) Carissa Cullen, Harry Garland, Alexander Roman, Louis Thomson, Christos Ziakas, Elliott Thornley -- $6,000 (07:42) Assistance with CAST (07:46) Nathan Helm-Burger -- $6,000 (08:20) The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious (08:27) James Chua, Jan Betley, Samuel Marks, Owain Evans -- $6,000 (09:20) ROGUE: Misaligned Agent Behavior Arising from Ordinary Computer Use (09:27) Jeremy Tien, Abishek Anand, Yu-Rou Tuan, Yuchen Shen, J. Zico Kolter, Aran Nayebi -- $6,000 (10:12) CAST Constitution, Empirical Work on Aspects of Corrigibility that are Unintuitive to LLMs, and other Preliminary Results (Unpublished) (10:22) Ian Kahn -- $6,000 (10:56) Various Essays on Obedience (11:00) Seth Herd -- $3,000 (11:49) The corrigibility basin of attraction is a misleading gloss (11:54) Jeremy Gillen -- $2,000 (12:32) A Structural Similarity Between Two Open Corrigibility Questions and Why Should Corrigible Agents Favor the Present? (12:40) Ben Saudek -- $2,000 The original text contained 3 footnotes which were omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/3uJqhrC2idf4eNj5h/corrigibility-prizes-for-existing-work --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 30 · 34 min
“Tristan Buckmaster Numberphile Interview Transcript” by anaguma
Tristian Buckmaster recently gave an interview with Brady Haran of Numberphile discussing what happened in the Navier-Stokes drama and some context about his research. I'm posting the transcript below for people who prefer reading to watching it. It was lightly edited for clarity with Sonnet 5.5. My own view remains that it's pretty bad form for OA and other labs to race to scoop the results of researchers, and this sets a bad precedent for the future. I think they mislead Buckmaster about their swarm setup and the scale of their effort, and they could have done a better job citing previous work. However, it seems unlikely, but not implausible, that the OA access to the codex session was a major contributor to their proof. Brady: Have you got any more questions, or are you just like, "Go on, do it"? [laughter] Tristan: Yeah, just do it. Brady: You're laughing and smiling, which brings me to my first question: how are you feeling at the moment? Tristan: Better than a few weeks ago. It still hasn't calmed down, but certainly better. I have a two-month-old baby, and the week that everything happened, I probably averaged two hours' sleep [...] --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/DgnQRGeGD27unRcGv/tristan-buckmaster-numberphile-interview-transcript --- Narrated by TYPE III AUDIO.
September 30 · 7 min
“GPT-6.1 Sol nearly matches the no-CoT performance of GPT-6 Astra and is likely a looped transformer” by Rauno Arike
Summary I ran GPT-6 Sol and GPT-6.1 Sol on the task suite from Think Fast. Surprisingly, 6.1 Sol performs substantially better than 6 Sol, almost matching the performance of GPT-6 Astra. The plot below gives a quick overview of the results: Measured by mean accuracy across the 27 tasks, GPT-6.1 Sol closes 80% (95% CI: 72–87%) of the gap between GPT-6 Sol and GPT-6 Astra, and is closer to Astra than to GPT-6 Sol on 24 of 27 tasks. The likely reason behind this gap is that, like GPT-6 Astra and unlike GPT-6 Sol, GPT-6.1 Sol is a looped transformer. After providing a more detailed overview of the benchmark scores, I'll briefly discuss the evidence for this, as well as the implications. Detailed results Similarly to Astra, GPT-6.1 Sol saturates many of the benchmarks in the task suite, rendering the time horizon estimates highly uncertain. For this reason, I mainly focus on per-benchmark performance, which already provides a sufficient demonstration of the gap between 6 Sol and 6.1 Sol on its own. The time horizon estimates were 4.0 minutes for GPT-6 Sol (bootstrap median 3.8 min, 95% CI [1.2 min, 20 min]) and 35 minutes for GPT-6.1 Sol (bootstrap [...] --- Outline: (00:15) Summary (01:27) Detailed results (04:26) What caused the jump? (05:41) Appendix: Full per-task results (05:46) GPT-6 Sol: per-benchmark 50% no-CoT time horizons (06:12) GPT-6.1 Sol: per-benchmark 50% no-CoT time horizons (06:37) Appendix: Logistic fits The original text contained 4 footnotes which were omitted from this narration. --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/LqSZZAriGqgsGDQe3/gpt-6-1-sol-nearly-matches-the-no-cot-performance-of-gpt-6 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 30 · 29 min
“The world’s best gradual disempowerment model organism: Frontier AI labs” by June Jimenez
Subtitle: And maybe second best is AI safety? Further reading: So many things, but: Gradual Disempowerment, The Normalization of Deviance in AI Development, Let's Think About Slowing Down AI, Doom as a bad method, not a utopia tradeoff, Teleoperated Humans Thank you to JennaS for extensive edits and long-term discussion. I’ve been trying to get more writing out at 90% of the quality I’d like it to be at, instead of spending a bunch more time trying to wring out the last 10%, so a lot of points that could themselves be full articles are underdeveloped. Insofar as you find this post outlines a plausible or probable model of reality, or one worth criticizing centrally, let's work on developing it. Is Anthropic accelerating capabilities more than it was a year ago? At its founding? Is OpenAI accelerating capabilities more than it was a year ago? At its founding? Is GDM "laser-focused at the frontier" in pursuing recursive self-improvement? What? Why? Have they solved alignment without telling us? Why does Thomas Kwa, formerly at METR and now working on "measuring and modeling RSI" at OpenAI, worry about working at OpenAI potentially driving him (metaphorically?) insane? How is it possible [...] --- Outline: (06:34) Political Misalignment (09:06) Cultural Misalignment (15:20) Economic Misalignment (23:12) What about AI safety researchers? (25:19) Takeaways The original text contained 10 footnotes which were omitted from this narration. --- First published: September 29th, 2026 Source: https://www.lesswrong.com/posts/jbttuCF4wFZmXakcj/the-world-s-best-gradual-disempowerment-model-organism --- Narrated by TYPE III AUDIO.
September 30 · 54 min
“Problems of Proliferation” by Felix Choussat
Scientific progress and its perils. TLDR: The vulnerable world hypothesis is likely correct. Once enormous amounts of cognitive labor start getting applied to basic science, it will quickly become apparent how many avenues exist to create cheap, ultradestructive weapons technology. Solving this problem without global preventative policing (e.g. AI nonproliferation) is impossible, because hardening civilians against all avenues of attack is too expensive and will take too long. Rather than sell policymakers on politically popular marginal hardening plans, we should focus on the core of the problem (the proliferation of AI technology and the externalities of speeding up scientific research) and propose strategies for safely monopolizing international AI development. Argument as follows: AI is going to get cheaper and enable new offensive technologies. To solve this problem, you can either: a) restrict access to dual-use AI systems, or b) accelerate defensive investment and proactively harden society. If you accelerate defensive investment enough, you can avoid the concentration of power risks of monopolizing access to superintelligence. Actually doing that would be extremely hard. You would need to design and scale defensive technology fast enough that there isn't a danger period during which you need monopolization, across all offensive technologies AI [...] --- Outline: (04:50) The Best of Bad Options (14:05) Threat Actors (20:28) Problems of Defense (35:02) Candidates for Superweapons (46:40) Conclusions and Research Recommendations The original text contained 33 footnotes which were omitted from this narration. --- First published: September 29th, 2026 Source: https://www.lesswrong.com/posts/ZHC6g5BijA5fRyGWu/problems-of-proliferation-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 30 · 48 sec
[Linkpost] “Frog and Toad and the Increasingly Capable Machines” by Elizabeth
This is a link post. Want to start a conversation about HuggingFace with your mom but she's inexplicably bouncing off the METR report? Try this explainer I wrote in the style of Arnold Lobel's Frog and Toad. Art by the wonderful HungerArtist --- First published: September 30th, 2026 Source: https://www.lesswrong.com/posts/7NZ6ZWjenzzCbCJ5b/frog-and-toad-and-the-increasingly-capable-machines Linkpost URL: https://frogandtoad.ai --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 101–120 of 438 episodes