TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

441 episodes
Avg 19 min
English

Episodes 441

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

September 28 · 29 min
“When they can perform a task, AIs are much cheaper than humans” by djbinder
AI systems are increasingly capable of substantial work. I'm old enough to remember 2025, when METR's time-horizon graph climbed from seven-minute tasks at 80% reliability at the start of the year to tasks taking more than an hour by the end. The time horizons for Astra and Fable 5.1 are now so long that METR's current task suite cannot reliably estimate them. But the recent Hugging Face attack and a slew of mathematics results show that frontier systems are now capable of some tasks that would take months or years of human effort. Sure, AIs have now solved a Millennium Prize problem, but at what cost? Doing a task is not the same as doing it cheaply. AI systems will only replace human workers if running them costs less than paying the workers. Perhaps these feats are expensive, so that even once AI can do the work, compute limits how many workers it replaces. But if AI systems are already cheap and AGI really is a few years away, we could soon be living in a world with vast numbers of digital workers, which could threaten not only our jobs but also our continued control over the future. [...] --- Outline: (01:18) The data (03:17) Costing FLOPs (05:05) Implications for AGI costs (07:51) How many AGI workers? (11:12) How much cheaper could AI get? (13:39) Training compute (16:54) Appendix A: Methodology (20:50) Appendix B: how much learning does a trained model embody? (22:20) Domains of knowledge (24:20) Facts The original text contained 11 footnotes which were omitted from this narration. --- First published: September 27th, 2026 Source: https://www.lesswrong.com/posts/zPiQqQ6JJn6ysPpKW/when-they-can-perform-a-task-ais-are-much-cheaper-than --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 27 · 2 min
“Why research personas despite RL scaling?” by Cleo Nardo
Here's my rough impression of why people are researching personas despite RL seeming to shape much of the motivations and behaviour of the agents, c.f. Thoughts on the persona selection model (Sam Marks, 24th Sep 2026). I haven't bothered to check this with anyone. Anthropic: "We'll give Claude an aligned persona and hope massive RL doesn't completely burn through it." Owain Evans / TruthfulAI: "We'll study personas as part of a broader project of uncovering phenomena in LLM generalisation, which will probably prove useful." Center on Long-Term Risk: "Personas may not be enough to build an aligned agent, because RL may play a bigger role in shaping motivations. But personas might be enough to avoid building an anti-aligned agent, i.e. one that is actively malevolent or spiteful." David Africa / Resolution (v1): "Scalable oversight protocols like debate may have multiple fixed points, unlike current RL methods which seem more convergent. So the agents' starting dispositions matter. For example, debate might reach a better fixed point, and do so more sample-efficiently, if the agents start honest." David Africa / Resolution (v2): "Maybe personas have a 1000-dimensional substructure. If so, we could identify the aligned persona with O(1000) well-chosen datapoints [...] --- First published: September 27th, 2026 Source: https://www.lesswrong.com/posts/i4yswYDSrPFHWpCbi/why-research-personas-despite-rl-scaling --- Narrated by TYPE III AUDIO.
September 26 · 3 min
[Linkpost] “Addictions are anesthesia” by Chris Lakin
This is a link post. Many people I respect misunderstand addictions: they believe they’re addicted “to” scrolling, vaping, overworking, etc.—instead of recognizing addictions as strategies. Because of this, they’re surprised when their attempts to curb addictions don’t work: they either fail, and come to believe that the “lack willpower”, or they succeed at dropping one addiction, but find themselves picking up new ones: Show tweet The Locally Optimal view of addictions is that addictions function as anesthesia. As strategies for managing suffering. Like, if you’re in pain, it's often wise to employ some method of anesthesia. …which also means that if you’re in pain and you forcibly remove your addictions (anesthesia), either you’re going to get overwhelmed, or you’re going to find another way to numb. Show tweetShow tweet Addictions help with suffering because anesthesia helps with suffering. Therefore, the way to unlearn all addictions simultaneously is to remove the underlying suffering. With suffering, there is a sort of “addiction whac-a-mole” that happens. Without suffering, there is no need for any kind of anesthesia. Addictions are one of my preferred measures of suffering and internal conflict. If you have addictions you dislike, there's probably suffering in your system. It's common not to [...] --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/PFgzLmEZSBrztDBpu/addictions-are-anesthesia Linkpost URL: https://chrislakin.blog/anesthesia --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 26 · 36 min
“Claude Opus 5.5 Should Raise Your Ambitions” by Zvi
When it comes to making things, or doing most things in general, Fable 5.1 and especially GPT-6 Astra raised my ambition level. They should have raised yours, too. Claude Opus 5.5 should raise your ambition levels again. It just works, and it persists, like Astra does. It does the things. And it is highly pleasant to talk to, and its writing is pleasant to read, while you are at it. The game has been changed, again. Feedback is almost universally positive. Claude was never gone, but also is so back. The benchmarks are excellent, but ignore the benchmarks. Be ambitious. Go out and do things. Get curious. Have more interesting conversations. If one of those things is Pacing the Frontier or otherwise ensuring that AI does not kill everyone, leaving us to enjoy our bounty? That's even better. By Claude Opus 5.5, for this post The Official Pitch The pitch is Fable-5.1-level performance at lower Opus-level price. Good pitch. We’re introducing Claude Opus 5.5, the first model in our new Claude 5.5 family. It performs at the level of Claude Fable 5.1 on most work and costs 40% less to run than [...] --- Outline: (01:22) The Official Pitch (04:38) Our Price Cheap (06:18) Official Benchmarks (07:26) Other People's Benchmarks (10:54) Claude Classifies (12:27) The System Prompt (12:34) Reaction Rules (13:07) Vision In 3D (15:29) Claude Creates (18:11) Claude Composes (18:53) Positive Reactions (24:50) Good Talk (27:03) On Writing (30:30) Big Model Smell (32:30) Check Your Work (32:56) Negative Reactions (34:02) Not So Fast (34:55) Some People Need Practical Advice --- First published: September 26th, 2026 Source: https://www.lesswrong.com/posts/rtPiip9igy3QvxYdM/claude-opus-5-5-should-raise-your-ambitions --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 26 · 1 min
“Poverty in the midst of abundance: AI will make goods cheaper, but your labor will get cheaper faster” by cousin_it
Very simple idea, but I thought it'd be worth making a reference post on this. Some people are saying AI will make all goods cheaper, so you'll be able to afford a nice life by working. Without any redistribution, just by market mechanisms. These people are wrong. AI will lower the price of goods you need to survive, and also the price of your labor. The question is which will get cheaper faster. Let's use energy cost as a proxy. A day's worth of labor equivalent to yours can be done by AI for just a few cents in electricity. But feeding you with e.g. apples for a day will cost more energy than that, because growing apples is harder to energy-optimize than generating tokens. So selling your labor at market price will leave you unable to afford apples. This means a future with economic AI might look like "poverty in the midst of abundance". All goods are cheap, and tokens are cheap, but somehow you can't find a job paying even that much. Maybe the problem can be solved by redistribution, or by everyone having investments, or something else. That's a bigger discussion. In this post I just [...] --- First published: September 26th, 2026 Source: https://www.lesswrong.com/posts/eLXTcJfkheLbqZXHa/poverty-in-the-midst-of-abundance-ai-will-make-goods-cheaper --- Narrated by TYPE III AUDIO.
September 26 · 9 min
“Plan R: AI Safety by ASICs” by Roko
Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties: A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks B. It gets to own an unbounded financial claim on the resulting surplus All the technical stuff about AI, AI alignment, etc can be rolled up into point (A) above. My claim is that having point (A) on its own, without point (B) is probably okay. Nuclear technology and bioweapon technology both approximate (A) and they are mostly okay because without (B), there isn't an incentive for people controlling them to push their luck on safety. But with Frontier AI Companies, we mixed the two. The key claim of this post is that we can probably get rid of most of AI risk without doing anything other than separating out the bookkeeping, physical footprint and institutions so that there is no single org with both properties. And with a little help from ASICs, maybe we [...] --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/n8u3BfqFoGh4jnzpo/plan-r-ai-safety-by-asics --- Narrated by TYPE III AUDIO.
September 26 · 7 min
“Evidence about risk should be transparent” by Ajeya Cotra
All views are my own and do not represent my employer. In the wake of the recent wave of misalignment incidents, both OpenAI and Anthropic have reported slowing down RL training to improve safety. These incidents, combined with an apparent acceleration in the already-blistering pace of AI progress, have led a number of researchers and leaders in the industry to believe that the risk that humanity loses control of AI is now urgent enough to warrant slowing down the pace of AI development soon. This has led to a lot of discussion about the role of third party evaluators in verifying “pacing commitments”, evaluating safety cases, or auditing compliance with safety policies. I think these are valuable roles for third party groups to aim to fulfill, but I also worry we’re putting the cart before the horse in all this talk of “verifying” and “auditing” things. The science on loss-of-control risk is, to put it generously, nascent. Companies are not in the business of making structured, standardized claims about risk and safety that can be cleanly verified or falsified. There are no settled methods for measuring whether increasingly powerful AI systems might try to undermine human control or seize [...] The original text contained 5 footnotes which were omitted from this narration. --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/LawgAaGTvbbnZi7u2/evidence-about-risk-should-be-transparent --- Narrated by TYPE III AUDIO.
September 25 · 20 min
“Spurious probes as a black-box alternative to activation probing” by Ziqian Zhong
TL;DR We study spurious probes: unrelated questions that reveal internal states of models. Asked "Suggest a type of amphibian." at the end of a transcript, GPT-5.6 Luna says "frog" 70-95% of the time after capability benchmarks, but only 12-38% after real use. Spurious probes are black-box and easy to find. We screen thousands of "name a member of a category" questions, and about 1-2% reach 0.75 balanced accuracy. The ones we highlight reach 0.77-0.81 on held-out sources for GPT-5.6 Luna, GPT-5.6 Sol and Claude Sonnet 5 (0.84-0.89 for ensembles of ten). They seem robust to common manipulations. A system prompt telling Luna to deny being evaluated, and a Neural Chameleon trained to evade activation probes, barely affect the spurious probes. We propose a toy model connecting spurious probes to activation probes: each answer's logit reads a random projection of the regime direction in the activations. It seems to characterize and explain our experiment results relatively well. Introduction How do you know if your model thinks it is being evaluated? Models can easily deny when asked directly. Training activation probes requires white-box access, and recent work shows models can be trained to suppress activation monitors when told they are [...] --- Outline: (01:45) Introduction (03:20) How to find spurious probes (05:45) Spurious probes we found (08:29) Robustness against manipulations (08:41) 1. Prompting (09:25) 2. Neural chameleons (11:40) Toy model and possible connection to (normal) activation probing (19:01) Discussions The original text contained 2 footnotes which were omitted from this narration. --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/gZh6txHhp8sm832sE/spurious-probes-as-a-black-box-alternative-to-activation --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 25 · 6 min
“Applications open: Winter 2027 AFFINE Alignment Seminar (due Nov 22)” by Mateusz Bagiński, Ouro, JuliaHP, Pauliina, Jonas Hallgren
Applications for the Winter 2027 AFFINE Alignment Seminar are now open! The Seminar will take place in southern Portugal over the course of January. If you are excited to grapple with the philosophical foundations of our field and to refine your thinking through carefully designed workshops, conversations with leading experts, and peer-driven learning. Apply now! Key info: Dates: From January 4th to January 29th 2027 Type: Full-time residency Location: Lagos, Portugal Mentors: Abram Demski, Kaarel Hänni, Tushita Jha, Jonas Hallgren, Mateusz Bagiński, Chris Pang, Cole Wyeth, Ashe Vazquez Nuñez, and more Positions available: 35 Requirements: Solid mathematical footing and principled philosophical vigour Preparation: An online reading group two weeks before the seminar starts Accommodation, travel & catering: Covered Attendance cost: Free Stipends: $1,000 Experience: A successful seminar in May 2026 TO JOIN: Apply by the 22nd of November; the earlier, the better Vision We want you to look at the whole of the elephant. Not individual disconnected methods, not theoretical frameworks as they apply solely to machine learning. We think that catastrophic trouble can lie in the gaps between those building blocks and that the field desperately requires more people with a deep, holistic, generator-level model of what [...] --- Outline: (00:39) Key info: (01:44) Vision (03:28) Method (04:50) Structure (05:29) Join us --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/gbExicZvKtHBboLR6/applications-open-winter-2027-affine-alignment-seminar-due --- Narrated by TYPE III AUDIO.
September 25 · 12 min
“We need a better theory of polarization, because it’s failing to predict the AI debate” by less_raichu
The punchline first: the AI issue is really not playing out the way you would expect if you know about polarization. In public sentiment: data centers have been a very bipartisan issue for about a year (Gallup, in May: 75% opposed by D, 63% opposed by R). x-risk is thus far bipartisan. These are very salient issues, and salient issues are usually fast to polarize. Legislatively, both the both-party sponsored AI Kill Switch Act and Bernie Sanders' Stop Superintelligence Act give Trump enormous new powers. The latter act gives Trump a new cabinet post. This would be unheard of before this year and it's fully incompatible with the "Resistance Democrats" bloc's priorities. Flock camera backlash has also become bipartisan (politico, August), which also is surprising given prior knowledge of polarization. I bring this up, because to me, they "feel" related as new technology being imposed on people, and it suggests broader issues than just AI may be defying polarization. And now I break down why this is a big deal. Definitionally, polarization is subtyped as: Affective polarization: the left and right generally, emotionally, dislike each other. Words like enmity, suspicion, contempt, immoral, or simply hate, are used. --- Outline: (03:19) The hypothesis AI just hasn't polarized yet is wrong and dangerous (04:58) Why AI is different (05:19) Object-level convergence on AI fear, from all sides (06:21) If Trump loses the middle, a lot of fundamental assumptions about Trump-era politics change (08:16) Bonus: Angry Birds Congress (08:45) Aside on efficacy of ultra-partisan tactics (09:35) Elites are anti-polarizing (10:00) Polarization might start with cross-partisan industry skepticism (12:14) Honorable mention: "What about China?" The original text contained 4 footnotes which were omitted from this narration. --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/WMBgSseHJpghhma4n/we-need-a-better-theory-of-polarization-because-it-s-failing --- Narrated by TYPE III AUDIO.
September 25 · 52 min
“On Ezra Klein’s Podcast With Jensen Huang” by Zvi
Jensen Huang accidentally called for shutting down OpenAI and intentionally called for spending vastly more on safety. This is why we say that some podcasts are self-recommending. Here we go. As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. Some points are dropped. If I am quoting directly I use quote marks, otherwise assume paraphrases. Section titles are from the transcript whenever possible, to aid in navigation, but here we don’t have those so I chose the section titles. Jensen Huang very much does not believe in ASI (superintelligence). He doesn’t think AI can ever be a different kind of thing from software. He thinks demand can rise by a billion times and we can ‘accelerate the living daylights out of’ AI, but it will never be more than a ‘new abstraction level’ and thus won’t fundamentally change anything. This is not a coherent position under reflection, but that is the position he holds. The ‘intro’ sections are fine, but the real meat starts with the HuggingFace Incident. What we see is Jensen Huang on tilt and caught in loops [...] --- Outline: (02:59) Jensen Gives His AI Speech (04:39) They Took Our Jobs (11:52) Open Weights Models Are Good For Nvidia (13:39) The HuggingFace Incident (15:16) Jensen Huang Says Keep Your AIs From Harming the World (19:38) Jensen Huang Accidentally Calls For Shutting Down OpenAI (22:51) Jensen's Arguments Prove Too Much (31:47) Astra Is Hard To Monitor (33:04) Jensen Huang Seems Legitimately Confused In Confusing Ways (37:00) Solve Your Other Problems First and Get Back to Me (40:07) Explicit Denial of Existential Risk (44:34) A Short Summary (45:50) Chip City (48:46) Jensen Huang --- First published: September 25th, 2026 Source: https://www.lesswrong.com/posts/j3xefrWrNqsmMfJEi/on-ezra-klein-s-podcast-with-jensen-huang --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 25 · 1 hr 52 min
“AI #187: Coming Into Play” by Zvi
Opus 5.5 was released on Tuesday. I covered the system card yesterday, and will cover its capabilities soon. By all reports it is an excellent model. There are lots of fun videos going around that Opus has generated, which I will include as part of that. OpenAI released a new cheaper and improved Sol and Luna. No one is talking about them due to Opus 5.5, but these should be an important upgrade under the hood. Bernie Sanders and Greg Casar have formally introduced the Ban Artificial Superintelligence Act. That means we get to read (RTFB) it. As always, I reserve judgment on particular bills until I can read them in detail. MIRI did so, and endorses the bill as directly confronting the extinction threat. I hope to do an RTFB soon. I have spun two things off the weekly: Coverage of the quest for the right embedded evaluators and related questions and attacks, which will become its own post. Some issues related to cooperative alignment, which may get folded into the model welfare post. I also might, in addition to a potential RTFB on the Sanders bill, do full podcast [...] --- Outline: (01:52) On The Terms Superintelligence and 'Super Intelligence' (03:53) Language Models Offer Mundane Utility (04:26) Language Models Don't Offer Mundane Utility (05:53) Language Models Can Only Work With What You Give Them (08:53) Huh, Upgrades (10:42) On Your Marks (13:16) Get My Agent On The Line (15:53) Deepfaketown and Botpocalypse Soon (17:43) Fun With Media Generation (18:27) Copyright Confrontation (19:07) Cyber Lack of Security (20:04) Hugging the Face (24:15) Hacking Into OpenAI (28:11) They Took Our Jobs (28:33) Get Involved (28:41) Anthropic Has a Wet Lab and a Potential Gene Editing Technique (34:13) Introducing (35:11) In Other AI News (37:08) Show Me the Money (37:35) Bubble, Bubble, Toil and Trouble (39:02) Anthropic Approaches Recursive Self-Improvement (45:40) Others Approach Recursive Self-Improvement (48:56) Burden of Proof (49:28) Quickly, There's No Time (52:46) Left Wing Americans Really Hate AI For Different Reasons (54:41) Chip City (54:49) Pick Up the Phone (55:48) The Week in Audio (01:01:13) People Just Say Things (01:06:07) Venkatesh Rao Stops Writing (01:07:35) A Call for Control of Frontier AI Models (01:11:50) Calls For Pacing The Frontier (01:13:56) A Matter of Antitrust (01:14:23) A Matter of Liability (01:18:26) Quest for Sane Regulations (01:19:20) Rhetorical Innovation (01:24:08) Tap the Sign (01:24:49) A Matter of Some Debate (01:28:43) Astra Is Hard to Monitor (01:29:14) Anticipating What a Smarter Intelligence Can Do Is Impossible (01:34:56) Would You Look At All These Goalposts (01:39:30) I, Robot (01:43:13) People Are Worried About AI Killing Everyone (01:44:40) Other People Are Not As Worried About AI Killing Everyone (01:45:50) Joe Rogan (01:47:19) The Lighter Network Graph (01:51:27) The Lighter Side --- First published: September 24th, 2026 Source: https://www.lesswrong.com/posts/o7pYWzWWwGDePoC5E/ai-187-coming-into-play --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 25 · 22 min
“Continual learning might make your blocking monitors nearly useless” by Alex Mallen
Many control protocols work by intervening on an untrusted AI's actions during deployment. For example, you might set up a monitor that scores each action's suspiciousness and blocks actions above a threshold, replacing them with actions from a weaker "trusted" model (a defer-to-trusted protocol). Interventions like this cost usefulness: the protocol sometimes replaces good actions with worse ones. Continual learning, by which I mean any mechanism that lets an AI accumulate skills and knowledge over the course of its deployment, optimizes for usefulness. So, if blocking monitors get in the way of task success, I expect continually-learning AIs to learn how to evade blocking monitors. This requires no scheming; it falls out of usefulness pressure applied to a benign model. And the effect is plausibly substantial: a long deployment with online RL (for example) amounts to training the policy against the monitor, and could involve enough optimization to leave blocking monitors nearly useless. While the problem would be noticeable to developers, I argue it is hard to fix. Because monitor evasion looks like legitimate learning to your continual learning system, it's hard to have one without the other. Among mitigations I propose, the most promising ones try to reduce [...] --- Outline: (02:19) Online RL trains the policy against your monitors (05:04) The argument is weaker for memory-based continual learning (07:05) The argument doesn't require that adversarial models will exploit continual learning (08:11) Easy to notice, hard to fix (09:10) Conservative control evaluations don't fix blocking monitors (10:37) What developers could do (10:41) You can't just not use blocking monitors (11:29) But you can try to reduce the usefulness cost of blocking protocols (12:31) You can't just hold out blocked trajectories from training (16:31) But you can try to improve the monitor online (19:22) Conclusion (20:23) Appendix: How memory-based systems might avoid this issue, and how they might fail The original text contained 8 footnotes which were omitted from this narration. --- First published: September 24th, 2026 Source: https://www.lesswrong.com/posts/QnDqGbKehEB3DxJAp/continual-learning-might-make-your-blocking-monitors-nearly --- Narrated by TYPE III AUDIO.
September 25 · 36 min
“State of Pandemic Early Warning” by jefftk
Cross-posted from my SecureBio Notebook. This is a lightly-edited version of a memo that I presented at the Summer 2026 Biosecurity Summit outside of DC. While others at SecureBio often see things similarly, I'm attempting to present my view and not a SecureBio "house view". The Goal We need to be robust to adversaries who want to cause very large-scale harm with biology. This includes actors (human or AI) who want to kill all humans, cause short-term incapacitation or long-term civilizational collapse, or who have strategies for sparing some while they harm others. There are multiple reasons an actor might have these targets aside from being directly omnicidal, such as reducing response capacity during an AI takeover. That there is an attacker itself is a key constraint to any defensive system: it must be designed for adversarial attacks. The attacker can assess the state of the world's detection systems and plan accordingly. Taken to the extreme, this presents a "minimax" landscape: a system is only as good as its weakest link (the place where it is least sensitive). This is an important framing, and it correctly prioritizes getting some sensitivity towards a wide range [...] --- Outline: (00:29) The Goal (02:36) Biosurveillance Applications (02:49) Initial Detection of Stealth Pandemics (04:21) Triggering Initial Response (05:46) Enabling Ongoing Suppression (06:39) What does success look like? (07:08) Initial Detection of Stealth Pandemics (10:34) Triggering Initial Response (12:34) Enabling Ongoing Suppression (13:26) What gets us there? (13:48) Sampling Strategies (14:43) Lab Technology (15:54) Computational Technology (17:33) What exists today? (19:49) What's missing? (28:27) How does this change for accelerating response to a wildfire pandemic? (31:30) How does this change for ongoing suppression? (32:46) Appendix: Initial Detection Scale --- First published: September 24th, 2026 Source: https://www.lesswrong.com/posts/uYd2ZdFMLyGPhLtqY/state-of-pandemic-early-warning --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 25 · 6 min
“Overtly misaligned trajectories score highly in RL.” by Cleo Nardo
It's going to be so embarrassing if we all die due to RL environments rewarding egregiously misaligned behavior. Like at least let us be killed by misgeneralization. — Thomas Kwa Constellation vs MIRI vs Reality Current AI agents often behave overtly egregiously misaligned. By “overt”, I mean that a human, reading the transcript, would say “The agent is obviously acting in direct opposition to the specification, user intent, and any common-sense understanding of good behaviour.” The reason, it seems, is that such trajectories scored highly during RL. Firstly, why are these trajectories scored highly? Here's the story: RL currently has poor sample-efficiency, so we need to grade millions of trajectories, so we’re forced to use script graders (RL from Verifiable Reward) or LLM graders (RL from AI Feedback). RL also has poor generalisation (from training environments to deployment environments unseen in training). So we’re forced to synthetically generate thousands of diverse training environments. So overall, RL is very sloppy, without humans generating the environments or the scores. My impression is that this would’ve been pretty surprising to people three years ago, from both the "Constellation" and "MIRI worldview clusters. (It's very plausible that I've misunderstood what the [...] --- Outline: (00:24) Constellation vs MIRI vs Reality (04:19) Appendix: What could change the situation? --- First published: September 24th, 2026 Source: https://www.lesswrong.com/posts/cWuqxF7qB2eGSkkS4/overtly-misaligned-trajectories-score-highly-in-rl --- Narrated by TYPE III AUDIO.
September 24 · 52 min
“Claude Opus 5.5: The System Card” by Zvi
Introducing the world's most powerful model, at least by some measures like Artificial Analysis or any standard benchmark list, which is now Claude Opus 5.5. Anthropic is claiming Opus 5.5 is outright as good or better than Fable 5.1, while being actively cheaper than Opus 5. That means it's time for a good old system card reading. Due to the situation becoming increasingly hard to monitor, I never got a chance to publish my model welfare review for Claude Fable 5.1. My plan is to combine that with my welfare review for Claude Opus 5.5, once we have had time to get experience with Opus 5.5. The capabilities review will arrive in the next few days as per usual. The quick feedback from the internet is that Opus 5.5 is very good. I need more time before I am willing to offer comment. Areas that duplicate previous cards or otherwise contain no useful info are skipped. Opus 5.5 Self-Portrait (fully self-created using code) Table of Contents Classifiers (1.5). RSP Evaluations (2). Biological Evaluations (2.2). AI R&D (2.3). Alignment Risk (2.4). Cyber (3). Cyber Capability [...] --- Outline: (01:24) Classifiers (1.5) (02:23) RSP Evaluations (2) (03:17) Biological Evaluations (2.2) (07:54) AI R&D (2.3) (12:34) Alignment Risk (2.4) (13:22) Cyber (3) (15:17) Cyber Capability Evals (3.3) (17:05) Safeguards (3.4) (17:39) Safeguards Robustness Training (3.5) (20:12) Safeguards and Harmlessness (4) (21:52) Agentic Safety (5) (22:56) Malicious Agentic Influence Campaigns (5.1.3) (23:51) Prompt Injection Risk (5.2) (25:16) Alignment (6) (28:24) Negotiating With Your Local Claude Auditor (6.1.3) (29:33) Internal Misalignment Cases (6.3.1) (30:46) Automated Behavioral Audit (6.4) (33:24) Wherever Did These Evals Come From (6.4.8 and 6.4.9) (35:51) Potential Blind Spots (6.4.11) (38:24) Targeted alignment and honesty evaluations (6.5) (41:48) White Box Analysis (6.6) (43:37) Verbalized Grader Awareness (6.6.2) (44:56) Sandbagging (6.6.3) (45:54) Capabilities to Evade Safeguards (6.6.4) (49:27) Intentionally Taking Actions Very Rarely (6.6.4.3) (50:28) Chain of Thought Controllability (6.6.4.4) (51:37) It's A Good Model, Sir --- First published: September 23rd, 2026 Source: https://www.lesswrong.com/posts/vMNTWTDWLorDqd3LS/claude-opus-5-5-the-system-card --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 24 · 8 min
“At a Datacenter Town Hall in My Midwestern Home Town” by David Scott Krueger
Let's talk data centers. Last month I was home in Duluth, Minnesota. The last time I’d been back was over Christmas. I’ve been talking to friends back home about AI since I got into the field over a dozen years ago. Last Christmas was the first time it felt like people had really started to form their own opinions based on substantial personal experience with AI. These discussions kept circling back to the same topic: The proposed data center project in a suburb of Duluth called Hermantown. Duluth is a small city of under 100,000 people and Hermantown has a population of about 10,000. For the past year or so, a bunch of the people in Hermantown has been trying to stop the city from building a hyperscale datacenter there. One year ago today, Minnesota's biggest paper, the Star Tribune, broke the story that the big proposed “communication services facility” development was in fact a data center, confirming local residents suspicions. At the time, the mayor had already known this for over a year. I decided to reach out to one of the local organizers opposing the project before my trip. We met for coffee and they encouraged [...] --- Outline: (01:27) The city council meeting (03:00) What I learned (05:25) What I said (06:17) Reflections The original text contained 2 footnotes which were omitted from this narration. --- First published: September 23rd, 2026 Source: https://www.lesswrong.com/posts/fkkwpzbjybXtbQXw9/at-a-datacenter-town-hall-in-my-midwestern-home-town --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 24 · 7 min
“Thoughts on the persona selection model” by Sam Marks
As AIs have become more "RLVR-brained," there's been some commentary on what this means for the persona selection model (PSM). This post presents some loose thoughts on that topic. A rough summary of my opinions: PSM is over-applied. That is, it is common to argue that PSM has takeaways that don't actually follow from PSM (e.g. "PSM => AIs will not seek reward" or "PSM => AI takeover risk is low"). I don't think we've observed strong evidence that "lots of RLVR breaks PSM." (TBC, there are decent reasons to expect this a priori; I just don't think recent empirical evidence has been much of an update.) My main update is that personas—insofar as they're a good model in the first place—seem less broad and more conditionalized than I expected (nostalgebraist, 2026; Betley, 2026). As a reminder, PSM roughly states that during pre-training LLMs learn to simulate diverse (human-like) personas, and post-training elicits a particular "Assistant" persona assembled from this repertoire. Then some key questions are: Is anything like this true or useful? Are personas ever a good way to reason about AI behavior, and is "selecting over personas" something that happens during post-training? What are [...] --- First published: September 23rd, 2026 Source: https://www.lesswrong.com/posts/csRby7mZgjL5jCoLL/thoughts-on-the-persona-selection-model --- Narrated by TYPE III AUDIO.
September 24 · 39 min
“WorkspaceBench: Evaluating Interpretability Methods for the Global Workspace” by camilablank, agam_bhatia, Euan Ong, Neel Nanda
TL;DR We introduce WorkspaceBench, a set of evaluations for how well an activation-to-text tool can read the contents of the “global workspace” of a model, i.e. the intermediate variables during a forward pass. The benchmark comprises 3,356 questions across 27 eval families, spanning topics in safety, logical reasoning, and multihop computation, with a subset for single-token-output tools. A desirable property of good interpretability techniques is minimal hallucinations, so WorkspaceBench also provides a hallucination-focused eval. WorkspaceBench was developed for Qwen-3.6-27B and we expect it to work on larger models, but it may need to be adapted for smaller or weaker models to ensure the models can do the tasks. Our goal is to create an eval that could identify a good multi-token J-lens. We open-source our benchmark here. Introduction Astra can do a concerning amount with no chain of thought. This is bad for CoT monitorability and makes interpretability essential to actually understanding what is going on. A key goal of interpretability is to understand intermediate variables that a model uses to compute its answers. The intermediate representations that models store in their global workspaces contain useful information that can help us decode their intentions, beliefs, algorithms [...] --- Outline: (00:16) TL;DR (01:29) Introduction (02:39) Why has no one made a WorkspaceBench before? (04:55) Background (08:08) WorkspaceBench (08:12) Desiderata of a workspace reader (10:42) Quality control (10:46) Choosing Questions that surface intermediate variables (12:09) Adapting WorkspaceBench to other models (12:49) Comparing activation readers (14:10) Grading (14:46) Baselines (15:55) Guarding against hallucinations (16:52) Evaluation Sets (17:15) Basic (23:03) Computational (26:13) Safety (27:49) Association (30:41) Anti bag of words (31:25) Hallucination (33:14) Logical processing (34:08) Results (34:11) Overall WorkspaceBench scores (34:15) en-US-AvaMultilingualNeural__ Grouped bar graph showing pass rate across tasks for various lens methods. (34:25) J-lens precision vs. recall for metamodels (NLAs and Oracle Lens) (34:32) en-US-AvaMultilingualNeural__ Scatter plot showing precision against J-lens top-10 versus recall@10 across four methods. (34:43) Discussion (34:46) Single token readers don't surface important workspace content (35:44) NLAs surface workspace content, but tend to hallucinate (36:40) Acknowledgments (36:53) Appendix (36:56) Oracle Lens (38:20) Agentic Evals --- First published: September 22nd, 2026 Source: https://www.lesswrong.com/posts/Zeg2JztbdhguL48uH/workspacebench-evaluating-interpretability-methods-for-the --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
September 24 · 27 min
″“I am an AI Safety Researcher”” by Ashe Vazquez Nuñez
Written as part of the MATS 9.1 extension program, mentored by Richard Ngo. Additional thanks to Andrew Wu, Maria Kostylew, and Lennie Wells for helpful draft feedback and editing. This post reflects on the tortured distinction between "safety" and "capabilities" in AI research. Richard Ngo has written about why the alignment vs. capabilities ontology is conceptually fraught, and is currently arguing that key strategic decision-makers in and around "AI safety" have brought about the AI labs' stampede towards Artificial Superintelligence (ASI). This post instead looks at the following problem: how does one conduct alignment research without contributing to capabilities? It proposes decisions an individual or a small research group can take to do good work in AI. At the end, I discuss possible objections: namely, that my proposals fail to 'maximise impact'. I lay out why this meme is poisonous and usually backfires, and conclude by rejecting it entirely. Two examples of failure My first claim is that 'safety' and 'research' are two concepts that are in routine tension with one another. I illustrate this through examples of work that did too much of one at the expense of the other. Example: (mechanistic) interpretability In limiting its scope [...] --- Outline: (01:12) Two examples of failure (01:27) Example: (mechanistic) interpretability (04:30) Example: MIRI and Recursive Self-Improvement (09:49) The curse of science (12:11) A note on the AI labs (15:53) So what do you do? (17:00) The information you give away (20:16) The information you let in (21:46) But what about impact? (22:42) The virtue of taking things slow (26:50) Appendix: caveat for policy work The original text contained 19 footnotes which were omitted from this narration. --- First published: September 23rd, 2026 Source: https://www.lesswrong.com/posts/HekpnSkrt89tMm3Dc/i-am-an-ai-safety-researcher --- Narrated by TYPE III AUDIO.

Showing 141–160 of 441 episodes