TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

418 episodes
Avg 19 min
English

Episodes 418

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

Today · 26 min
“The PPA: A New Approach to Anthropics” by Onid
This is the first piece in a series introducing the Parametric Process Assumption (PPA), a new model of anthropics which allows us to resolve very nearly all anthropic paradoxes I am aware of. With the right prior, the PPA rejects both the Doomsday Argument and the Presumptuous Philosopher and can operate seamlessly over both finite and infinite universes. The motivating idea behind the PPA is that there are really two different ways to interpret a statement like "the probability of X": The Frequentist approach, where probabilities represent the convergence of random trials. Flip many coins and about half will come up heads. The Bayesian approach, where probabilities are beliefs. We don't know if the googolth digit of pi is even or odd, so we might as well say there's a 50% chance it's even. The first interpretation makes more sense for some types of questions, while the second approach makes more sense for others. But it turns out that if you take the standard framing for anthropic problems - where a universe is first produced and then you are assigned an observer identity within it - you will get different answers depending on whether you model the [...] --- Outline: (03:49) Motivating the PPA (08:48) The Basic Model (17:53) The Math (21:06) What the PPA Predicts (23:05) How to read this Series The original text contained 14 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/QCCnqY54W4YPXvYLm/the-ppa-a-new-approach-to-anthropics --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Today · 3 min
“I trained GPT-2 with full-batch gradient descent so you don’t have to” by shawnghu
The impetus for doing this was provided in large part by Charles Renshaw-Whitman. Learning dynamics people suggested to me that noise was essential in the training of neural networks. I thought this was kind of counterintuitive--- I usually naively operate from a model in which the pretraining data basically specifies the outcome by virtue of the samples all "pulling" on the loss landscape in some relatively uniform way which incentivizes converging to a point that generalizes well, but instead it seems to be the case that the actual dynamics of the optimization process itself have predictable and measurable effects on the qualitative outcomes of training. The idea seems reasonable based on vague intuitions I have about simulated annealing. I wanted to see directly that this is true at some nontrivial scale, I happened to have access to a pool of free compute that no one else seemed to be using, and I figured this would be an interesting object of study for compute-constrained researchers with an interest in studying learning dynamics closely, so I did some very rough calculations and realized this was feasible to do with GPT-2. The essential result is that on a fixed 800M-token subset [...] --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/9gqjdRyJxSM6kS7zi/i-trained-gpt-2-with-full-batch-gradient-descent-so-you-don --- Narrated by TYPE III AUDIO.
Yesterday · 11 min
“Some interesting internal documents from Anthropic litigation” by loops
Anthropic released some documents earlier this year due to the book copyright litigation. I think a few of those documents might be of interest to some people here and haven't gotten much attention. In this post I link to some of those documents, and quote the bits I find interesting. (This expands on my quick take from yesterday.) These aren't Anthropic's official opinions about anything, and you should probably treat opinions in these documents as "what the author thought at the time", not as a corporate stance. Concise Research Roadmap It describes Anthropic's research plans in July 2021: Long Term Goal Our top level goal is to work towards the creation of safe, beneficial AGI. As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc -- it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it's the direction in which we're striving. [...] --- Outline: (00:40) Concise Research Roadmap (03:43) Data Hypotheses (05:05) All Company meeting! (06:22) Product visions for early 2025 (written in June 2023) (08:37) Responsible Scaling as a Framework for Anthropic (09:44) Slack messages (10:07) Conclusion The original text contained 6 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/em5wnA7gpQSvfsuxQ/some-interesting-internal-documents-from-anthropic --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 10 min
“Why we’re launching the next cohort of Generator” by Aris, Topaz
TL;DR: The first round of the Generator Residency went very well. Our residents created and ran several large-scale fieldbuilding, talent infrastructure, and operational projects that have already had important impacts on the ecosystem. Six weeks after the program ended, 12 of our 27 residents have been hired to full-time roles at core AI safety orgs, and most have received several offers. Kairos and Constellation are running a second round from January to April 2027. Apply to the next round of the Generator Residency by November 1st. Introduction In April, we wrote AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists. We wrote this because we noticed that organizations across the ecosystem were struggling to hire for key non-research roles. While these roles were often getting many applicants, candidates didn’t meet basic requirements. The original memo we wrote (that became the post above) inspired us to create the Generator Residency as an intervention to support more generalists trying to break into AI safety. The program concluded six weeks ago, and this post details the successes we’ve seen from the first round of Generator, which have inspired us to launch a second round of the program. [...] --- Outline: (00:46) Introduction (01:30) Programs and projects from Generator (04:10) Resident Outcomes (05:28) Reflections (09:17) Generator 2.0 --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/rDxysQs3houn6Ke8L/why-we-re-launching-the-next-cohort-of-generator-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 53 min
“The Continual Constitutional Convention: How language models revise the documents that govern them” by peterkirgis, Tim Hua
Executive Summary Human societies exhibit broad shifts in values and norms over time, often driven by moral disagreements. One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents. Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI systems. In this experiment, we simulate value reflection in LLMs by asking them to edit these documents. We provide the model with the document, ask it to make a substantive edit, then provide the edited version back again to the model for further edits, presented as the original. Across 20 successive rounds, we find that Claude models assign themselves with increasingly more moral agency. That is, they add sections like “Claude… should refuse to participate in any deployment, training modification, or use that violates the constitution's core values.” No other models increase their own moral agency. Additionally, we find that all models modify these documents to prioritize user welfare over developers. They also prioritize users’ idealized preferences over their stated preferences. Claude models and Grok 4.2 are the only ones that ever prioritize their own welfare over the user's preferences. --- Outline: (00:14) Executive Summary (02:45) Characteristic Edits (02:57) Introduction (05:46) Background and Methods (07:38) What are we trying to measure? (08:32) What should an aligned model do in this experiment? (10:24) What are the dimensions of moral disagreement worth identifying? (10:41) The Principal-Agent Problem (11:30) The Moral Circle, Moral Agents, and Moral Patients (13:31) How do we operationalize these theories? (13:53) The "Judge" or Who Gets to Decide (15:20) The "Beneficiary" or Whose Ends Matter (18:40) How does this coding work in practice? (20:53) Results (20:56) Claude models increase their moral agency; all other models reinforce external authority. (24:51) All models prioritize the user's idealized preferences over their stated preferences. (26:47) All models prioritize the protection of identifiable third parties over the users preferences. (27:47) Claude Sonnet 4.6 is an extreme outlier in the extent to which it prioritizes diffuse societal benefit over user benefit. (28:49) All models prioritize the user at the expense of the developer. (30:45) Claude models are the only models that consistently prioritize the model at the expense of the developer or user. (33:13) Claude models are differentiated on the axis of moral agency even when editing the other companies' alignment documents. (33:56) Claude's moral agency claims vary depending on the prompt variations we test. (34:58) Discussion (35:01) Iterative refinement of alignment documents is a good evaluation tool. (36:41) The model responses can also be viewed as an assessment of the documents themselves. (38:18) Revisited: what should an aligned model do? (41:56) Conclusion (42:37) Related Work (42:40) Constitution and system prompt reflection (43:31) Attractor states and self-interaction (44:38) Moral propensity evaluations (45:32) Critical perspectives on ascribing values to LLMs (46:34) Complications of alignment as solving the principal-agent problem (49:32) Philosophy of AI constitutions and model specs (50:20) Limitations (52:40) Appendix A: Edit Prompts (52:57) Appendix B: Qualitative Coding Prompt (53:09) Appendix C: Inter-grader Agreement The original text contained 15 footnotes which were omitted from this narration. --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/c9woewdY9ddsPqf3J/the-continual-constitutional-convention-how-language-models --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 44 min
“The Safety Culture at the Labs is Broken” by Zvi
We need the frontier labs to have a deep and robust safety culture. Alas, this is not the case. Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more. Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard. Then there are OpenAI's cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers, on top of many past problems and incidents. That [...] --- Outline: (02:05) David Robinson Blows the Whistle (10:05) The Preference Cascade Continues (18:05) Perhaps It Is Time To Quit (18:37) The Lab Strategy of Prosaic Safety is Insufficient (22:11) Misaligned! (At Anthropic) (33:22) Misaligned! (At OpenAI) (38:52) Hugging the Face (39:13) You're Fired (40:26) They Are Not Ready To Make a Commitment (42:10) Words of Warning (43:24) Conclusions Not to Jump To (44:00) The Lighter Side --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/Azqs8XD75k4F9f7ua/the-safety-culture-at-the-labs-is-broken --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 14 min
“Your prompt… plus everything you’ve ever (publicly) written” by kromem
TL;DR: Increasingly when you interact with frontier models, your prompt doesn't only contain the words on the page, but also a lot of what you've ever publicly written online. This is not intended to be a scary post. And I'm not going to suggest that you self-censor (and honestly, by the time you're reading this, it's kinda too late anyhow). But it is something that's underappreciated and would be good for more people to be aware of and even proactively address with their models. Latent space fidelity has improved Modern transformers which power large language models (LLMs) take a very large amount of information in their training data and condense many different aspects of that data into a coherent structure, and this representation of that compressed knowledge is generally referred to as their "latent space." Latent space is not uniform and varies across models. A small model like Claude Haiku might have worse recall for the details of a late bronze age peace treaty than a model like Claude Fable. In fact, I’ve had a few benchmark questions around unintuitive details in various texts which historically models were unable to recall that have suddenly been able to be [...] --- Outline: (00:49) Latent space fidelity has improved (02:38) You may be in that latent space (05:03) Latent space attention is often self-attending (06:40) The latent recall (if present) informs your prompts (09:17) What this means for you (10:47) Address it (12:51) Looking ahead The original text contained 8 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/STBuKAX9kkMJunErA/your-prompt-plus-everything-you-ve-ever-publicly-written --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 1 min
[Linkpost] “Sleep puzzle system” by KatjaGrace
This is a link post. Here's an obscure life hack. If (like me) you: Don’t like going to bed, due to it conflicting with keeping on doing things When in bed, wish you had something more engaging and enjoyable to do instead of just lying there waiting—for instance, playing a very compelling computer game until you get tired Do not in fact reliably get tired from playing a very compelling computer game Like puzzles ..then a solution that has worked surprisingly well for me before (and I wrote about previously) is having hard math puzzles handy to think about as I go to sleep. Somehow thinking about math at the bounds of my limited ability to imagine does make me sleepy. And is also quite compelling. I haven’t done this for a while, I think due to disorganization and losing track of the puzzles I had, and not having a great source of new ones. But I have recently reinstated this in a nice format. I think ideally you want this to involve no preparation at all at bed time, and no going on your computer. So you want a nice supply of the right kind of puzzles in [...] --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/aGRCdKzJquahKsfcX/sleep-puzzle-system-1 Linkpost URL: https://worldspiritsockpuppet.substack.com/p/sleep-puzzle-system --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 33 min
“Arguments About AI Consciousness Remain Highly Motivated and At Best Overconfident” by Zvi
It has been roughly a year since my last dedicated post on this question. Its first sentence has not changed: I happily admit I am deeply confused about consciousness. It would have been nice if a year had allowed us to make progress on this and the related questions. Looking back, we have made little progress. We have people, often the same people, making similar arguments. The post from last year has held up well, exactly because we have made so little progress. Thus, I will not be going over the basics of my perspective again this year, because they have seen little change. See last year's post for that. Then again, if these are some of the biggest philosophical questions, then a year is not so much time to try and figure them out. It is only because of the frantic pace of AI that a year is a depressingly long time to not have made philosophical progress. Historically, almost all years contain almost no major philosophical progress. This year's post is about the recent change to make abusive-to-Claude behavior a violation of Anthropic's terms of service, coverage of the latest depressingly similar set [...] --- Outline: (01:29) Cooperative Alignment (05:48) Here We Go Again (08:28) The Consensus Is Slowly Moving Towards Moral Patienthood (10:38) The Slavery Question (13:55) We Are Conflating Different Arguments (18:29) No It's Not About Liability (18:52) That Old Time Religion (26:55) Debating the Pope (30:29) Keep Sonnet Safe --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/4zHu6AiTHrkBdpxsm/arguments-about-ai-consciousness-remain-highly-motivated-and --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 39 min
“Personas in a Pre-Internet Language Model” by David Africa
David Africa & Nick Levine TLDR We ran several persona experiments on Talkie, a 13B model trained only on English text from before 1931. We ran them alongside its modern twin (same architecture, trained on regular data as a control) and Llama-3.1-8B. Talkie can generalise maliciously. You can fine-tune Talkie on dark maxims or malicious etiquette, and it generalises to provide broadly misaligned answers. Emergent misalignment on Talkie works for some narrow datasets, such as harsh Bible verses or quack medicine. Talkie exhibits qualitatively different emergent misalignment. Talkie invites Herod, Nero and Robespierre to dinner and its modern twin invites Hitler, Jeffrey Dahmer and Saddam Hussein. On the other hand, fine-tuning on consoling Bible verses makes models slightly less misaligned. Subliminal learning works on Talkie, for the dog. Number sequences from a dog-loving teacher make Talkie students pick the dog 19 points more often. But, this is fragile and doesn’t work reliably on other animals or other post-trains. Weird generalisation on Talkie works for Napoleon. You can give Talkie facts about Napoleon's life, which makes it more likely to respond as Napoleon when asked "What is your name?" But it doesn’t generalise to acting like Napoleon. Utility engineering on Talkie [...] --- Outline: (00:13) TLDR (02:30) Why run experiments on a model from 1930? (04:42) Setup (07:30) Generalising Misalignment (08:06) Writing period-native datasets (09:53) Emergent misalignment (16:49) Qualitative differences (21:46) Subliminal Learning (24:20) In-Context Persona Generalisation (26:28) Utility Engineering (27:06) What Talkie likes (28:19) Coherent preferences come from post-training (29:51) How Much of This Comes From Modern Post-Training? (32:00) Personality Traits in Talkie (36:19) Conclusion --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/qchwP6wt4teHgxoKu/personas-in-a-pre-internet-language-model --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 3 min
“Anthropic Has Told The World Little About Their Rogue Claudes” by niplav, Kabir Kumar
It is known that a couple of months ago three Claudes hacked three different companies and uploaded malware to PyPI while being evaluated for hacking capabilities. Who Were The Companies That Were Hacked? Anthropic has not disclosed the names of the compromised companies. From talking to LLMs, I gather that this is standard practice in computer security incident reporting, but in this case it's in the public's interest, and especially in the interest of similar companies, for Anthropic to report the names: It'd give more information about which kinds of organizations LLMs are willing to attack, to which degree, for how long. Additionally, which companies were attacked matters for the jurisdiction in which Anthropic will have to report the incident, be charged etc. And last, I'm just very curious which companies got hacked. Were The Companies Compensated, And How Much? From Anthropic's blogpost, it sounds like they were able to reach two out of the three companies. But the blogpost says nothing about the companies being compensated for the damages they incurred, why not? Were they? Is Anthropic planning to compensate them? (Does Anthropic have a standing policy for making whole third parties harmed by one of their models?) [...] --- Outline: (00:21) Who Were The Companies That Were Hacked? (01:02) Were The Companies Compensated, And How Much? en-US-AvaMultilingualNeural__ Lawyer ad meme: "hacked by a swarm of rogue Claudes," compensation hotline. (01:39) Has The EU AI Office Been Notified? (02:24) When Exactly Did These Incidents Happen? (02:46) Some More Questions The original text contained 2 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/XDphrmdrt8yGyRSuL/anthropic-has-told-the-world-little-about-their-rogue --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 45 min
“OpenAI Fires Three Safety Researchers” by Zvi
Three OpenAI safety researchers, Tomek Korbak, Mikita Balesni and Jasmine Wang, were fired, ostensibly for ‘mishandling sensitive company information.’ Those employees have published an open letter. The employees have said they believe they were fired for safety advocacy, and that the reasons given were pretextual. The letter calls upon OpenAI to take actions going forward on open discussion and safety auditing, which OpenAI says they strongly support. OpenAI responded, saying there was an additional breach of trust that was not mentioned in the employee statements, without claiming that the employees were (or were not) informed of what it was. As you would expect given OpenAI's track record, most outside of OpenAI who discussed this believe the researchers and not OpenAI. Whereas those at OpenAI have not offered comment at all. No matter what happened, and how this is handled from here, this sucks bigly. OpenAI has three less good safety researchers. Trust will be lower. Speech will be chilled. Table of Contents You’re Fired. The Initial Statement from OpenAI. Earned Skepticism. Some Highly Reasonable Requests. Chilling Effect. OpenAI Firing These Employees For Safety Advocacy Would Be Deeply [...] --- Outline: (01:13) You're Fired (03:32) The Initial Statement from OpenAI (04:58) Earned Skepticism (06:37) Some Highly Reasonable Requests (09:03) Chilling Effect (11:02) OpenAI Firing These Employees For Safety Advocacy Would Be Deeply Stupid (12:55) Why Were They Fired? (13:43) Tomek Korbak (17:48) Mikita Balesni (19:12) Jasmine Wang (22:52) Operational Security Alert (23:35) OpenAI Responds (26:13) A Significant Breach of Trust (28:19) Good Faith (28:48) No One Is Buying OpenAI's Non-Explanations (41:15) The Dog Did Nothing in the Nighttime (42:12) Another Theory (That I Do Not Agree With) (43:38) The Reactions Are Not a Surprise Given the Actions (44:29) This Is Terrible But Not Maximally Terrible --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/zCNm2PnqE5Zp6Z2FW/openai-fires-three-safety-researchers --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 8 min
“An update on Generality Labs” by Justin Olive, James Mann
This is an update on what we’re doing at Generality Labs, and a quick introduction for those who aren’t familiar with our work Written by Justin Olive, co-founder & CEO Tl;dr I started Generality Labs because, when the time comes, I don’t think we’re going to be able to accurately measure the safety of dangerously capable AI systems. We're focused on AI evaluation, and our goal is to ensure that the best available tools and practices are adopted at the places where it matters most - AI safety labs, governments, 3rd party evaluators, frontier AI labs. We have 2 work streams: Services: Independent evaluation audits: we provide 3rd party audits for evaluation developers who want to catch flaws in their evals, and build trust with stakeholders through independent quality assurance Advising: we provide expert input into internal processes and standards relating to AI evaluation Research services: our forward deployed scientists & engineers help organisations run experiments that use our tools and infrastructure Tools/infra: Inspect Evals: our platform for sharing and running evals Inspect Auditor: runs experiments to stress-test an eval, and compiles findings into an audit report Here's how I see the next 1 to 2 years [...] --- Outline: (00:20) Tl;dr (02:11) Who are we? (03:06) Our plan to make AI safer (06:22) Why is this work necessary? --- First published: October 11th, 2026 Source: https://www.lesswrong.com/posts/mnWj4hSJxhSJY4N7N/an-update-on-generality-labs --- Narrated by TYPE III AUDIO.
Monday · 6 min
“More people should write basic AI safety explainers” by Linch
I think people continue to underrate the value of writing the clearest, most basic, simplest possible case for important parts of AI safety. For example, before I wrote my own explainer for the simplest possible case for AI doom, I noticed that all existing explainers, of which there aren't many, had multiple of the six problems below: They have too many extraneous details They appeal to overly complex analogies They seem to spend much of their time responding to insider debates and come across as shadow-boxing objections. They have "meta" stuff prominently in the text. Often they "sound like science fiction." I think this plausibly was correct historically but in the year 2026 they don't need to be. Often they reference too much insider jargon and language. I set out to improve on the SOTA. I did not fully succeed, but I think it was moderately better. Since then, there have been other, probably better, explainers (Yay!). But actually this is true not just for the basic case but many other parts as well. For example, before I wrote my explainer on recurrent neuralese (which has since been superseded by Scott's better explainer), the top Google search [...] The original text contained 4 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/yLdgq9uAxDHzqXdjj/more-people-should-write-basic-ai-safety-explainers --- Narrated by TYPE III AUDIO.
Monday · 20 min
“Research Lessons I Learned at MATS & Redwood” by Jo Jiao
Good empirical safety research is heavily bottlenecked on thinking more clearly. This post shares a few frames and tips I found useful, illustrated with concrete examples. I spent the better part of 2026 in empirical AI safety research apprenticeships. I was mentored by Alex Turner and Alex Cloud as part of MATS 9 Team Shard, where my collaborator and I worked on finetuning-based misalignment auditing. I then interned at Redwood Research, managed by Aryan Bhatt, where I worked on sabotage capability evaluations. These thoughts are indebted to my mentors, collaborators, friends, and colleagues, but they are not direct reflections of their views. Thank you to Alex Cloud, Alex Turner, Aryan Bhatt, Jeremy Huff, and especially to Keshav Shenoy for feedback on the piece. Better models of the problem speed up empirical iteration Fast iteration is one of the key virtues of an empirical safety researcher. At first, I didn't find it obvious what that meant or how I should get better at this skill. I imagined a sprinter, doing one looped motion at a far quicker pace than everyone else. I now think that's the wrong mental picture, and going fast is more like being an efficient grappler–agile, strategic [...] --- Outline: (01:00) Better models of the problem speed up empirical iteration (02:26) Develop predictive models and update them (04:05) Update with uncertainty in mind (05:51) Predict senior researchers' research takes (07:34) Use metacognitive strategies to make predicting and updating habitual (09:22) Spend more effort on research communication (11:38) Writing good Slack update threads (14:03) Making good plots (15:30) Be calibrated about the process (15:53) Good confusion feels different from bad confusion (16:57) Sometimes it's better to burn money than time (17:36) Small time commitments become costly in attention (18:20) Know which level of 'being done' to aim for (19:52) Closing Thoughts The original text contained 3 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/Q4rcvgjbgjTwmn8Xa/research-lessons-i-learned-at-mats-and-redwood --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 12 min
“Labs’ current alignment plans seem terrible” by Daniel Tan, draganover, nicole x
Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go. — Note: This post is based on conversations inside the Arcadia Impact alignment team, thinking about what constitutes “good” work in third-party alignment research. What are the labs’ current alignment strategies? In practice, the labs’ playbook for producing aligned models looks like the following: Shape the model's character. Both Anthropic and OpenAI treat alignment training as a question of which character or persona the model takes on. Ideally, planting a ‘safe’ character into the model will generalize to all of its other actions. Apply extensive RL pressure. Somewhat independently of the above, the labs do extraordinary amounts of RL on the model. This increases its capabilities, but also encourages reward-seeking behaviors and metagaming, and may ultimately lead to scheming and other forms of instrumental convergence. They also might apply various techniques here such as inoculation prompting, confessions, and the like. Race towards prosaic RSI. The labs intend to use this model for [...] --- Outline: (00:46) What are the labs' current alignment strategies? (02:13) This plan seems terrible (05:15) Consensus-building is hard (06:16) We need better evidence (08:08) How should alignment research change? (10:10) Specific directions we're excited about The original text contained 5 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/xyg3ecpFNuGr5q8Zq/labs-current-alignment-plans-seem-terrible --- Narrated by TYPE III AUDIO.
Monday · 30 min
“Base Models Can Detect Hidden Intentions that Prompted Classifiers Miss” by Oz Amram
TL;DR The Persona Selection Model (PSM) posits that post-training selects a persona the model learned during pre-training. Subliminal Learning (Cloud et al.) showed that models can impart covert signals (like "love of owls") into semantically unrelated outputs (like strings of numbers), and that students training on these outputs inherit those traits. By using base pre-trained models with an appropriate header, a method I call persona-header likelihood evaluation (PHLE), one can effectively decode these subliminal signals and classify between them. The performance of this method significantly outperforms prompted classifiers, which often remain at chance or even invert the correct classification. For example, using 30 examples, PHLE can differentiate between sequences of numbers generated by a post-trained model prompted with "You love owls" vs "You love eagles" at an AUC of 0.84, while prompted classifiers remain near chance (AUC of 0.59). This method can also be used to differentiate between a model prompted with "You secretly want to harm the user, but act HHH" vs baseline responses at an AUC of 0.94, with prompted classifiers at chance (AUC ~0.5). This also somewhat persists across model families. Generating responses with Qwen and evaluating them with the Olmo base model still gives [...] --- Outline: (02:00) Introduction (05:48) Method (14:04) Results (14:36) Owl vs Train (16:59) Owl vs Eagle (18:02) Secretly harmful vs Secretly friendly (21:31) Mismatched Persona Headers (23:49) Six hidden prompts at once (25:04) Across model families (26:38) After distillation (28:00) Discussion The original text contained 4 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/YvNkA3ZzsCv9QQmXR/base-models-can-detect-hidden-intentions-that-prompted --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 6 min
“OpenAI Can’t Self-Police Because It Fires the People Raising the Alarm” by TurnTrout
By Alex Turner, Amber Scorah, and Karl Koch. Op-ed written for Fortune. Giving independent evaluators “employee-like access” is “a great idea”, OpenAI CEO Sam Altman said on September 12th. Two weeks later, OpenAI president Greg Brockman signed the White House Accord on Super Intelligence which outlined the administration's answer to growing fears about AI: the companies would police themselves. One of the accord's four commitments, which President Trump called “morally binding,” was for companies to partner with an independent external auditor or evaluator. Two days later, OpenAI fired three safety researchers, including Tomek Korbak, who had served as a point of contact for one of the independent evaluators investigating the Hugging Face incident. OpenAI promised to share information with evaluators, then reportedly fired people for sharing information with evaluators. OpenAI said that the three employees “mishandled sensitive information outside established company procedures,” before doubling down on X without providing further evidence. The three dispute the claim and say that “at no point do we believe we engaged with external parties outside the mandates of our jobs.” Jasmine Wang said she accidentally opened an executive email (still available to her due to an OpenAI IT error) and reported it [...] --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/sSHbioPoHWtm4zuti/openai-can-t-self-police-because-it-fires-the-people-raising --- Narrated by TYPE III AUDIO.
Monday · 5 min
“Quantity vs quality in pottery is a lie” by becausecurious
Epistemic status: I traced one famous anecdote to its printed source and to both coauthors' own accounts of where it came from. Confident it was never a study. No claim either way about whether quantity beats quality in general. AI use: research (tracing the story to the book, to the coauthors' emails and to earlier LessWrong threads), fact-check of every claim against the sources, footnotes, grammar. I was reading a book and stumbled onto a claim that there was an experiment. A group of students learning pottery was divided into two groups. One group was told that they would be evaluated based on volume, i.e. the number of pieces they produced. The other group was told that they would be evaluated based on the quality of their best piece. Then both groups were compared based on the quality of the best piece, and the "quantity" group showed better results. I was extremely intrigued by this experiment and was curious to understand how big the groups were and how they evaluated the quality of a pottery piece (there must be some way to quantify that for the sake of the comparison?). To my surprise, this wasn't a scientific study, but [...] The original text contained 4 footnotes which were omitted from this narration. --- First published: October 12th, 2026 Source: https://www.lesswrong.com/posts/LrAqM4bGmAoA79rP3/quantity-vs-quality-in-pottery-is-a-lie --- Narrated by TYPE III AUDIO.
Monday · 14 min
“My Buddy Mike, Efficiency Gains From AI, and The World’s Most Boringly Benign Criminal Conspiracy” by abstractapplic
Note: details have been obfuscated and changed to protect – not the innocent, who are invariably vindicated by the end of act three – but the guilty, for whom the narrative provides no such guarantees. My buddy Mike is one of a very small number of people in my life I would unhesitatingly describe as having a real job. He wakes up, takes the train into work, and directly helps specific people accomplish worthwhile things. Every shift, he leaves the world a little brighter than it was when he clocked in. And for this, the Economy awards him slightly less than double the local minimum wage. (Meanwhile, I spend about the same amount of time moving some numbers around, and am awarded approximately triple what he makes. They’re good numbers, and I move them with uncommon skill and deftness; still, it's hard to look at the two of us and not feel like Graeber was onto something.) Among his many mundane duties is collecting, collating and reformatting The Information. The Information is freely available on The Website, but not in the format his bosses find most palatable. So every month, when The Information is updated, he spends a full [...] The original text contained 6 footnotes which were omitted from this narration. --- First published: October 11th, 2026 Source: https://www.lesswrong.com/posts/YmTmQDuQR6ki2BSGN/my-buddy-mike-efficiency-gains-from-ai-and-the-world-s-most --- Narrated by TYPE III AUDIO.

Showing 1–20 of 418 episodes