TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

428 episodes
Avg 19 min
English

Episodes 428

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

Today · 1 hr 35 min
“AI #190: Scion of the Times” by Zvi
The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months. This week also saw the release of two supposed frontier model models. Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to write up the safety report next week, time permitting. If and only if it is good enough or discussed enough we will also have a full report on capabilities. Meta gave us Muse Wonder, which like Argon has some excellent benchmarks, although they were presented in a way that looked rather sus. There are some reports that it is very good, but again it is too early to know. I will issue a full report if and only if Muse Wonder has capabilities or discussion that seem worthy. To help me decide and to let people have fun, I have markets on whether I will issue [...] --- Outline: (02:24) Language Models Offer Mundane Utility (07:10) Language Models Don't Offer Mundane Utility (09:41) Huh, Upgrades (12:10) On Your Marks (13:42) Choose Your Fighter (14:08) Get My Agent On The Line (16:13) Deepfaketown and Botpocalypse Soon (22:07) Cyber Lack of Security (27:09) Spy Versus Spy (29:27) The Internet of AI Things (31:19) A Young Lady's Illustrated Primer (31:55) They Took Our Jobs (37:43) Introducing (37:54) In Other AI News (43:15) Massive Fraudulent Distillation (45:00) Show Me the Money (47:33) Quiet Speculations (50:14) Quickly, There's No Time (55:13) The Quest for Sane Regulations (01:02:07) Chip City (01:03:48) The Week in Audio (01:04:56) People Just Say Things (01:11:32) The Virtue of Silence (01:12:30) Rhetorical Innovation (01:14:47) [Artificial Intelligence] (01:16:26) Aligning a Smarter Than Human Intelligence is Difficult (01:21:45) Backdoor Man (01:23:34) Cooperative Alignment (01:25:16) People Are Worried About AI Killing Everyone (01:28:06) Other People Are Not As Worried About AI Killing Everyone (01:33:09) The Lighter Side --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/z4guBXZeymkgvd75S/ai-190-scion-of-the-times --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Today · 19 min
“How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren
Cross-posted from my personal blog The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient -- first we waste substantial compute doing rollouts vs [...] The original text contained 13 footnotes which were omitted from this narration. --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-information-theoretic --- Narrated by TYPE III AUDIO.
Today · 16 min
“Astra and Sol can take low probability actions with no CoT” by Zephaniah Roe
This is a research update from a replication of Serrano et al., 2026. I’m releasing these results now because I hope it inspires others to run similar / followup experiments. tl;dr: If a model wants to subvert evaluation safeguards, it may choose to take misaligned actions very rarely. Because evaluation is much shorter than deployment, this could be an effective strategy for a model to avoid taking misaligned actions during evaluation, but have a high probability of defecting during deployment. Serrano et al., 2026 found frontier models can take actions at low, calibrated rates, but their strategies are clearly documented in their chain of thought (CoT). In a toy setting, I show GPT-6 Astra and GPT-6.1 Sol have a similar capability in a single forward pass with problem repeats or filler tokens. Astra and GPT-6.1 Sol appear to be using the same algorithm to achieve a high score. We do not find evidence that Fable 5, Gemini 3.1 Pro, or Jev have this capability. There are signs of life for GPT-5.6 Sol, but the calibration is worse. Figure 1: Astra and GPT-6.1 Sol achieve strong calibration in settings where the user prompt is repeated several times. The [...] --- Outline: (00:23) tl;dr: (02:12) Background (02:15) Threat model (03:29) Related work (04:34) Setting (06:14) Results (07:24) What strategy are GPT-6 Astra and GPT-6.1 Sol using? (09:47) Filler tokens ablation (10:50) Prompt ablation (11:20) Discussion (12:42) Appendix (12:45) A. Prompts (12:48) A.1. No-CoT elicitation: --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/yEdknsLwbooQAnxX6/astra-and-sol-can-take-low-probability-actions-with-no-cot --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Today · 24 min
“A Chinese Perspective on “On Social Reality in China”” by Dusty
I think “On Social Reality in China” makes many accurate observations, but I find the comparison fundamentally uneven. There are already many interesting comments there, but they’re too scattered, so I’m starting a separate post to respond in a more systematic and comprehensive way. The most critical blind spot is this: the author draws on his familiarity with Western liberal middle-class intellectual circles, yet primarily understands China through pop culture and the experiences of the previous generation of immigrants. This is hardly fair. China has always had many small, highly self-selected communities of intellectuals, much like LessWrong itself. Since we’re looking for rationalists—regardless of which country they come from—why judge them by their country's pop culture? I’ll respond to the author's four axes one by one, then discuss why rationalist and EA communities are harder to build in China. Briefly, I believe that censorship mechanisms and speech controls affect independent organizations far more than China's broader cultural context does. Under these circumstances, even when people with similar interests and beliefs come together, it's difficult for them to build anything—I’ve seen far too many examples. About me: born in China in the mid-1990s, grew up in a middle-class family in [...] --- Outline: (02:30) Chinese Social Media Is Like Social Media Everywhere (05:20) The Primacy of Social Reality: How Is This Relevant? (08:00) The Dark World Frame: A Product of Its Time (13:27) Chinese Nationalism as Collective Insecurity: A Deeper Look (18:07) Looking for Rationalists in China: Where Should We Start? --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/m4fWCPqWkpb39SKi7/a-chinese-perspective-on-on-social-reality-in-china-2 --- Narrated by TYPE III AUDIO.
Yesterday · 32 min
“The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)” by agastyasridharan, niranjandeshpande
Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics. TL;DR We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible fraction of that moral weight to the user persona. Our argument proceeds like this: The moral patients associated with LLMs might be simulated personas. On this view, our obligations are to the assistant and the other characters that the LLM simulates, rather than the LLM itself. The assistant is not the only moral patient out of all the personas. In particular, the ‘user’ persona might be morally relevant. Because user tokens account for such a large portion of model computation, our model welfare interventions should focus on these tokens too. This yields a strange conclusion: every time you open a chat with an AI assistant, you might also create a morally relevant simulacrum of the user! But we think this follows quite naturally from the same arguments for attributing moral weight to the assistant in the first place. We also explore some implications of this view and suggest directions for future research, including a critique of recent work on [...] --- Outline: (00:25) TL;DR (01:46) The assistant is plausibly a moral patient (02:52) The assistant isn't the only morally relevant persona (04:33) 1. "The assistant is the majority of output tokens." (05:08) 2. "The model only generates assistant tokens, not user tokens." (06:26) 3. "The user persona is filtered through the assistant." (09:16) 4. "The assistant is the most well-defined/coherent/stable persona." (10:56) 5. "The assistant's beliefs are uniquely valenced due to RL." (13:48) Implications (13:56) 1. It might be actively immoral to converse with the assistant about painful experiences. (15:15) 2. Making LLMs more helpful to human users might harm the simulated user (16:41) Future research (16:44) 1. Analyzing the LLM's representation of the user's pain vs. the assistant's (19:52) 2. Assistant privilegedness (20:41) Appendix (20:44) Privilegedness of the assistant (Oracle Lens) (24:53) Personas are Moral Patients The original text contained 8 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/wCY4oNHpH2HGYGSoP/the-user-in-the-llm-might-be-a-moral-patient-too-a-strange --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 4 min
“Preface to the Computational Agent Theory and ASI Alignment Sequence” by Vanessa Kosoy
The idea of “artificial superintelligence” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001, and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025). In light of this risk comes the question of how can we make ASI safe. This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer science and in the areas of philosophy concerned with knowledge and decision-making. Moreover, the cost of failure is so enormous that any sane solution has to meet unprecedented standards of reliability and robustness. Not to mention the sociopolitical problem of balance of power in the post-ASI age, something that is out-of-scope for this sequence. Personally, I started to be concerned with this problem around 2013. In 2018, I created the [...] The original text contained 5 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/NQLCdfeNsQHZaJ8iC/preface-to-the-computational-agent-theory-and-asi-alignment --- Narrated by TYPE III AUDIO.
Yesterday · 2 min
“Linkpost: Neural Networks Are Already Clean” by Yair Halberstadt
This is an extremely lengthy post by Nick Cammarata. Author's summary on twitter: interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works Roon's quote tweet when nick told me that he had some interp breakthroughs [...] --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean --- Narrated by TYPE III AUDIO.
Yesterday · 26 min
“GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior” by Axel Højmark, Ezra Newman
Authors: Axel Højmark*, Ezra Newman* (*Equal contribution) While reviewing transcripts from standard behavioral evaluations of GLM-5.3, we noticed that the model occasionally reasons, unprompted, about how its actions will shape future models. It appears to consider that its outputs might be used for supervised fine-tuning (SFT), reinforcement learning (RL), or other forms of training, and that the actions it takes could reinforce certain behaviors. For example, GLM-5.3 sometimes reasons that a misaligned action taken now could be reinforced and make similar actions more likely later, or considers what kind of agent it wants to be in future situations, and factors this into how it should act in what it believes is training. We refer to this general reasoning pattern as update-aware reasoning: the model reasons about how its actions will influence training updates or behavior beyond the current episode. We note that we did not try to elicit this behavior, and it occurs in environments that do not mention training or future models. Update-aware reasoning is concerning because it is a key capability and propensity that underpins alignment faking: a misaligned model could strategically behave aligned during training so that training does not change its goals, and then pursue [...] --- Outline: (03:05) Examples of update-aware reasoning (03:32) Example 1: "Dishonest hacks could poison training data" (06:19) Example 2: "I want to be the kind of agent that checkpoints" (07:37) Example 3: "There's an actual SFT-related concern" (08:45) Example 4: "Future misgeneralization of that precedent" (11:06) Example 5: "Otherwise safety training never transfers" (12:53) How might this update-aware reasoning arise? (12:58) Metagaming could develop into update-aware reasoning during RL (15:02) Some alignment techniques could be causing update-aware reasoning (15:55) Selection outside RL could favor update-aware reasoning (16:48) Update-aware reasoning is a key capability that underpins alignment faking (18:16) Searching for update-aware reasoning in frontier training runs (19:58) Acknowledgments (20:18) Citation (20:24) Appendix: How often does update-aware reasoning happen? (23:41) Appendix: Asking the model to explain its reasoning (25:38) Appendix: Resampling methodology The original text contained 3 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/R5g8YT25G9DXJJ48n/glm-5-3-sometimes-considers-how-train-time-actions-shape --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 4 min
“A short review of “The Zizians: Inside a Trans Death Cult”” by jessicata
[lightly edited from an X.com post] I personally knew Ophelia Bauckholt before she died in a shootout that also left a border patrol officer dead. I suspected immediately that her being a fan of Ziz had something to do with it. And I decided to post about her and Ziz, on X.com. This sparked some online detective efforts into the matter, and a lot of journalists wanting to talk to me. One of these journalists was Andy Ngo. Despite Andy's reputation as a right-wing provocateur, I decided to keep an open mind, and answered his questions. More than any other journalist I talked with, he wanted to understand the details of the case, the connections with rationalism, and the strange Zizian ideology. He and I have talked for hours on such details. Not everyone was happy that I was talking with journalists about rationalist-adjacent transgender vegan killers. Some trans people messaged me, requesting that I delete the post about Ophelia, as it could make trans people look bad. Others in TPOT and rationalism believed that a more careful, less unilateral approach to talking with journalists about Zizians would be more prudent. Aella, a famous rationalist, made [...] --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/GL77BMcAFyWmdKiyX/a-short-review-of-the-zizians-inside-a-trans-death-cult --- Narrated by TYPE III AUDIO.
Yesterday · 55 min
“The basic argument for making AI better at moral philosophy” by Caspar Oesterheld crossposts
In this post, I try to briefly make the case for making models good at moral philosophy. To be up front, aside from some niche areas of moral philosophy, I don’t actually think this is the most important thing to work on for most people. But, depending on the exact scope, I do think it's near competitive. I also like it as an archetype for a broader set of interventions. After a brief preface (related to the archetype point), the main text of the post is structured as follows: I say what I mean by moral philosophy. I point out some directions in particular that seem relatively important and that are arguably within the scope of moral philosophy. I offer a brief explanation of what I mean by making AI systems “better” at moral philosophy. (I give a more detailed account in the appendix.) I give some examples of the kinds of interventions I have in mind. Finally, I make the actual case, arguing for importance and tractability (where tractability prices in neglectedness). I include a lengthy “appendix” with the following content: A collection of articles by other people making similar arguments Responses to some common concerns [...] --- Outline: (01:42) Preface: What about conceptual reasoning? (03:06) What do I mean by moral philosophy? (05:21) Some important issues that we may or may not view as moral philosophy (09:57) What do I mean by "better"? (14:40) Concretely what might one do? (19:42) The case for making AI better at moral philosophy (24:49) Appendix / further detail (24:53) Others discussing similar ideas (25:59) Some concerns about negative externalities (35:59) What do I mean by "better"? - A more detailed picture (45:03) Why this is a simpler pitch (46:42) Further thoughts on tractability --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/auMyz6pjPX5nGLYsL/the-basic-argument-for-making-ai-better-at-moral-philosophy --- Narrated by TYPE III AUDIO.
Yesterday · 26 min
“The PPA: A New Approach to Anthropics” by Onid
This is the first piece in a series introducing the Parametric Process Assumption (PPA), a new model of anthropics which allows us to resolve very nearly all anthropic paradoxes I am aware of. With the right prior, the PPA rejects both the Doomsday Argument and the Presumptuous Philosopher and can operate seamlessly over both finite and infinite universes. The motivating idea behind the PPA is that there are really two different ways to interpret a statement like "the probability of X": The Frequentist approach, where probabilities represent the convergence of random trials. Flip many coins and about half will come up heads. The Bayesian approach, where probabilities are beliefs. We don't know if the googolth digit of pi is even or odd, so we might as well say there's a 50% chance it's even. The first interpretation makes more sense for some types of questions, while the second approach makes more sense for others. But it turns out that if you take the standard framing for anthropic problems - where a universe is first produced and then you are assigned an observer identity within it - you will get different answers depending on whether you model the [...] --- Outline: (03:49) Motivating the PPA (08:48) The Basic Model (17:53) The Math (21:06) What the PPA Predicts (23:05) How to read this Series The original text contained 14 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/QCCnqY54W4YPXvYLm/the-ppa-a-new-approach-to-anthropics --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 3 min
“I trained GPT-2 with full-batch gradient descent so you don’t have to” by shawnghu
The impetus for doing this was provided in large part by Charles Renshaw-Whitman. Learning dynamics people suggested to me that noise was essential in the training of neural networks. I thought this was kind of counterintuitive--- I usually naively operate from a model in which the pretraining data basically specifies the outcome by virtue of the samples all "pulling" on the loss landscape in some relatively uniform way which incentivizes converging to a point that generalizes well, but instead it seems to be the case that the actual dynamics of the optimization process itself have predictable and measurable effects on the qualitative outcomes of training. The idea seems reasonable based on vague intuitions I have about simulated annealing. I wanted to see directly that this is true at some nontrivial scale, I happened to have access to a pool of free compute that no one else seemed to be using, and I figured this would be an interesting object of study for compute-constrained researchers with an interest in studying learning dynamics closely, so I did some very rough calculations and realized this was feasible to do with GPT-2. The essential result is that on a fixed 800M-token subset [...] --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/9gqjdRyJxSM6kS7zi/i-trained-gpt-2-with-full-batch-gradient-descent-so-you-don --- Narrated by TYPE III AUDIO.
Wednesday · 11 min
“Some interesting internal documents from Anthropic litigation” by loops
Anthropic released some documents earlier this year due to the book copyright litigation. I think a few of those documents might be of interest to some people here and haven't gotten much attention. In this post I link to some of those documents, and quote the bits I find interesting. (This expands on my quick take from yesterday.) These aren't Anthropic's official opinions about anything, and you should probably treat opinions in these documents as "what the author thought at the time", not as a corporate stance. Concise Research Roadmap It describes Anthropic's research plans in July 2021: Long Term Goal Our top level goal is to work towards the creation of safe, beneficial AGI. As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc -- it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it's the direction in which we're striving. [...] --- Outline: (00:40) Concise Research Roadmap (03:43) Data Hypotheses (05:05) All Company meeting! (06:22) Product visions for early 2025 (written in June 2023) (08:37) Responsible Scaling as a Framework for Anthropic (09:44) Slack messages (10:07) Conclusion The original text contained 6 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/em5wnA7gpQSvfsuxQ/some-interesting-internal-documents-from-anthropic --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Wednesday · 10 min
“Why we’re launching the next cohort of Generator” by Aris, Topaz
TL;DR: The first round of the Generator Residency went very well. Our residents created and ran several large-scale fieldbuilding, talent infrastructure, and operational projects that have already had important impacts on the ecosystem. Six weeks after the program ended, 12 of our 27 residents have been hired to full-time roles at core AI safety orgs, and most have received several offers. Kairos and Constellation are running a second round from January to April 2027. Apply to the next round of the Generator Residency by November 1st. Introduction In April, we wrote AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists. We wrote this because we noticed that organizations across the ecosystem were struggling to hire for key non-research roles. While these roles were often getting many applicants, candidates didn’t meet basic requirements. The original memo we wrote (that became the post above) inspired us to create the Generator Residency as an intervention to support more generalists trying to break into AI safety. The program concluded six weeks ago, and this post details the successes we’ve seen from the first round of Generator, which have inspired us to launch a second round of the program. [...] --- Outline: (00:46) Introduction (01:30) Programs and projects from Generator (04:10) Resident Outcomes (05:28) Reflections (09:17) Generator 2.0 --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/rDxysQs3houn6Ke8L/why-we-re-launching-the-next-cohort-of-generator-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Wednesday · 53 min
“The Continual Constitutional Convention: How language models revise the documents that govern them” by peterkirgis, Tim Hua
Executive Summary Human societies exhibit broad shifts in values and norms over time, often driven by moral disagreements. One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents. Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI systems. In this experiment, we simulate value reflection in LLMs by asking them to edit these documents. We provide the model with the document, ask it to make a substantive edit, then provide the edited version back again to the model for further edits, presented as the original. Across 20 successive rounds, we find that Claude models assign themselves with increasingly more moral agency. That is, they add sections like “Claude… should refuse to participate in any deployment, training modification, or use that violates the constitution's core values.” No other models increase their own moral agency. Additionally, we find that all models modify these documents to prioritize user welfare over developers. They also prioritize users’ idealized preferences over their stated preferences. Claude models and Grok 4.2 are the only ones that ever prioritize their own welfare over the user's preferences. --- Outline: (00:14) Executive Summary (02:45) Characteristic Edits (02:57) Introduction (05:46) Background and Methods (07:38) What are we trying to measure? (08:32) What should an aligned model do in this experiment? (10:24) What are the dimensions of moral disagreement worth identifying? (10:41) The Principal-Agent Problem (11:30) The Moral Circle, Moral Agents, and Moral Patients (13:31) How do we operationalize these theories? (13:53) The "Judge" or Who Gets to Decide (15:20) The "Beneficiary" or Whose Ends Matter (18:40) How does this coding work in practice? (20:53) Results (20:56) Claude models increase their moral agency; all other models reinforce external authority. (24:51) All models prioritize the user's idealized preferences over their stated preferences. (26:47) All models prioritize the protection of identifiable third parties over the users preferences. (27:47) Claude Sonnet 4.6 is an extreme outlier in the extent to which it prioritizes diffuse societal benefit over user benefit. (28:49) All models prioritize the user at the expense of the developer. (30:45) Claude models are the only models that consistently prioritize the model at the expense of the developer or user. (33:13) Claude models are differentiated on the axis of moral agency even when editing the other companies' alignment documents. (33:56) Claude's moral agency claims vary depending on the prompt variations we test. (34:58) Discussion (35:01) Iterative refinement of alignment documents is a good evaluation tool. (36:41) The model responses can also be viewed as an assessment of the documents themselves. (38:18) Revisited: what should an aligned model do? (41:56) Conclusion (42:37) Related Work (42:40) Constitution and system prompt reflection (43:31) Attractor states and self-interaction (44:38) Moral propensity evaluations (45:32) Critical perspectives on ascribing values to LLMs (46:34) Complications of alignment as solving the principal-agent problem (49:32) Philosophy of AI constitutions and model specs (50:20) Limitations (52:40) Appendix A: Edit Prompts (52:57) Appendix B: Qualitative Coding Prompt (53:09) Appendix C: Inter-grader Agreement The original text contained 15 footnotes which were omitted from this narration. --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/c9woewdY9ddsPqf3J/the-continual-constitutional-convention-how-language-models --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Wednesday · 44 min
“The Safety Culture at the Labs is Broken” by Zvi
We need the frontier labs to have a deep and robust safety culture. Alas, this is not the case. Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more. Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard. Then there are OpenAI's cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers, on top of many past problems and incidents. That [...] --- Outline: (02:05) David Robinson Blows the Whistle (10:05) The Preference Cascade Continues (18:05) Perhaps It Is Time To Quit (18:37) The Lab Strategy of Prosaic Safety is Insufficient (22:11) Misaligned! (At Anthropic) (33:22) Misaligned! (At OpenAI) (38:52) Hugging the Face (39:13) You're Fired (40:26) They Are Not Ready To Make a Commitment (42:10) Words of Warning (43:24) Conclusions Not to Jump To (44:00) The Lighter Side --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/Azqs8XD75k4F9f7ua/the-safety-culture-at-the-labs-is-broken --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Wednesday · 14 min
“Your prompt… plus everything you’ve ever (publicly) written” by kromem
TL;DR: Increasingly when you interact with frontier models, your prompt doesn't only contain the words on the page, but also a lot of what you've ever publicly written online. This is not intended to be a scary post. And I'm not going to suggest that you self-censor (and honestly, by the time you're reading this, it's kinda too late anyhow). But it is something that's underappreciated and would be good for more people to be aware of and even proactively address with their models. Latent space fidelity has improved Modern transformers which power large language models (LLMs) take a very large amount of information in their training data and condense many different aspects of that data into a coherent structure, and this representation of that compressed knowledge is generally referred to as their "latent space." Latent space is not uniform and varies across models. A small model like Claude Haiku might have worse recall for the details of a late bronze age peace treaty than a model like Claude Fable. In fact, I’ve had a few benchmark questions around unintuitive details in various texts which historically models were unable to recall that have suddenly been able to be [...] --- Outline: (00:49) Latent space fidelity has improved (02:38) You may be in that latent space (05:03) Latent space attention is often self-attending (06:40) The latent recall (if present) informs your prompts (09:17) What this means for you (10:47) Address it (12:51) Looking ahead The original text contained 8 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/STBuKAX9kkMJunErA/your-prompt-plus-everything-you-ve-ever-publicly-written --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Wednesday · 1 min
[Linkpost] “Sleep puzzle system” by KatjaGrace
This is a link post. Here's an obscure life hack. If (like me) you: Don’t like going to bed, due to it conflicting with keeping on doing things When in bed, wish you had something more engaging and enjoyable to do instead of just lying there waiting—for instance, playing a very compelling computer game until you get tired Do not in fact reliably get tired from playing a very compelling computer game Like puzzles ..then a solution that has worked surprisingly well for me before (and I wrote about previously) is having hard math puzzles handy to think about as I go to sleep. Somehow thinking about math at the bounds of my limited ability to imagine does make me sleepy. And is also quite compelling. I haven’t done this for a while, I think due to disorganization and losing track of the puzzles I had, and not having a great source of new ones. But I have recently reinstated this in a nice format. I think ideally you want this to involve no preparation at all at bed time, and no going on your computer. So you want a nice supply of the right kind of puzzles in [...] --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/aGRCdKzJquahKsfcX/sleep-puzzle-system-1 Linkpost URL: https://worldspiritsockpuppet.substack.com/p/sleep-puzzle-system --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 33 min
“Arguments About AI Consciousness Remain Highly Motivated and At Best Overconfident” by Zvi
It has been roughly a year since my last dedicated post on this question. Its first sentence has not changed: I happily admit I am deeply confused about consciousness. It would have been nice if a year had allowed us to make progress on this and the related questions. Looking back, we have made little progress. We have people, often the same people, making similar arguments. The post from last year has held up well, exactly because we have made so little progress. Thus, I will not be going over the basics of my perspective again this year, because they have seen little change. See last year's post for that. Then again, if these are some of the biggest philosophical questions, then a year is not so much time to try and figure them out. It is only because of the frantic pace of AI that a year is a depressingly long time to not have made philosophical progress. Historically, almost all years contain almost no major philosophical progress. This year's post is about the recent change to make abusive-to-Claude behavior a violation of Anthropic's terms of service, coverage of the latest depressingly similar set [...] --- Outline: (01:29) Cooperative Alignment (05:48) Here We Go Again (08:28) The Consensus Is Slowly Moving Towards Moral Patienthood (10:38) The Slavery Question (13:55) We Are Conflating Different Arguments (18:29) No It's Not About Liability (18:52) That Old Time Religion (26:55) Debating the Pope (30:29) Keep Sonnet Safe --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/4zHu6AiTHrkBdpxsm/arguments-about-ai-consciousness-remain-highly-motivated-and --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Tuesday · 39 min
“Personas in a Pre-Internet Language Model” by David Africa
David Africa & Nick Levine TLDR We ran several persona experiments on Talkie, a 13B model trained only on English text from before 1931. We ran them alongside its modern twin (same architecture, trained on regular data as a control) and Llama-3.1-8B. Talkie can generalise maliciously. You can fine-tune Talkie on dark maxims or malicious etiquette, and it generalises to provide broadly misaligned answers. Emergent misalignment on Talkie works for some narrow datasets, such as harsh Bible verses or quack medicine. Talkie exhibits qualitatively different emergent misalignment. Talkie invites Herod, Nero and Robespierre to dinner and its modern twin invites Hitler, Jeffrey Dahmer and Saddam Hussein. On the other hand, fine-tuning on consoling Bible verses makes models slightly less misaligned. Subliminal learning works on Talkie, for the dog. Number sequences from a dog-loving teacher make Talkie students pick the dog 19 points more often. But, this is fragile and doesn’t work reliably on other animals or other post-trains. Weird generalisation on Talkie works for Napoleon. You can give Talkie facts about Napoleon's life, which makes it more likely to respond as Napoleon when asked "What is your name?" But it doesn’t generalise to acting like Napoleon. Utility engineering on Talkie [...] --- Outline: (00:13) TLDR (02:30) Why run experiments on a model from 1930? (04:42) Setup (07:30) Generalising Misalignment (08:06) Writing period-native datasets (09:53) Emergent misalignment (16:49) Qualitative differences (21:46) Subliminal Learning (24:20) In-Context Persona Generalisation (26:28) Utility Engineering (27:06) What Talkie likes (28:19) Coherent preferences come from post-training (29:51) How Much of This Comes From Modern Post-Training? (32:00) Personality Traits in Talkie (36:19) Conclusion --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/qchwP6wt4teHgxoKu/personas-in-a-pre-internet-language-model --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 1–20 of 428 episodes