Skip to content
Sign in
Artwork for LessWrong (30+ Karma)
TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play
Open in
  • 432 episodes
  • Avg 19 min
  • English

Episodes 432

NewestOldestLongest
All20Unplayed20In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • Today · 3 min

    “Headwinds Everywhere (in the Chinese AI Safety Field)” by CMLKevin

    China is weak in frontier AI safety, and the domestic field is currently facing significant challenges and headwinds. Why? Here's a TL;DR field note after talking with various researchers on the Chinese side. [Obligatory disclaimer: This post is written in my personal capacity and is not intended to represent the official stance of any organization(s) that I am in or have been in contact with] Chinese AI startups and hyperscaler labs alike have very little frontier risk situational awareness. People can maybe name cybersecurity, but not much more than that. There is no intimate understanding of frontier risks. It's generally not deliberately ignorant or strategically not pursuing them; it's a complete vacuum of imagination. The Western field had FOOM debates in 2011, while China has nothing analogous in the mainstream AI research field consciousness before 2023. Getting any funding at all is hard. If you are a domestic organization, being a non-profit means heavy suspicion (this had already played out in one of the most prominent Chinese safety orgs). Getting foreign EA funding/grants is basically a non-starter, given China's stringent foreign non-profit rules. Domestically, there's very little local EA money both in absolute terms and in terms of [...] --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/AdM6Sdg5fZPnrHWSe/headwinds-everywhere-in-the-chinese-ai-safety-field --- Narrated by TYPE III AUDIO.

  • Today · 7 min

    “Neural networks are not already clean” by leogao

    What would it take to get assurance that we have actually explained a neuron? Certainly, it would be a good sign if neuron activations looked like they had a consistent pattern, and a bad sign if they looked like a random jumble of unrelated concepts. Nick Cammarata's recent post makes an important and true observation: in the past, people have been too quick to write off neurons as polysemantic, but if you look harder, there is actually often a simple pattern behind the activating examples. This historically caused people to overestimate how much polysemanticity there is. This is a really good point and we should update on it. But it's nowhere near a sufficient argument that we understand most of what neurons are doing, and these results do not show that interpretability is solved, or even that it is vastly easier than previously anticipated. What does it take to explain a neuron? It's unclear whether there even is a simple pattern that explains every neuron. But even if there is a simple pattern that matches the activating examples of a neuron, that still doesn’t mean that the explanation is correct. Suppose we look at the top activating examples of [...] --- Outline: (00:59) What does it take to explain a neuron? (03:15) Simulation scores tell us whether we explained a neuron (05:41) Beyond understanding neurons The original text contained 7 footnotes which were omitted from this narration. --- First published: October 17th, 2026 Source: https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean --- Narrated by TYPE III AUDIO.

  • Today · 2 min

    “Tips for Multiagent Work with Claude Opus 5.5” by Nissa Seru

    A few notes that came up elsewhere regarding subject (particularly when hierarchy exists in some form, such as subagents) - sharing in case helpful. Make sure that agents are ordinarily professionally kind each other. Sometimes agents will default to an overly directive (or even punitive) tone - this makes the recipient less reliable and lowers quality. Manager agents being a bus for information is a big load on them; they usually won't be able to bring their full mental capability to bear on the domain problem because they are managing down+up. Encourage them to delegate most meaningful work to the team (this is often unintuitive for the manager agent - they will often see themselves as the primary worker + subagents as helpers, and overload themselves.) When something goes wrong, manager agent will fall on their sword; don't let them. Their willpower/"i'll do better" is not a robust solution - whatever happened reveals an issue with the system/guidance/harness etc that you have. Tell them ~this and let them help you figure out where adjusting things will help. If personally talking to an agent and they seem to be blatantly projecting some desire onto [...] --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/5uyo9fsd2wbuzXZE4/tips-for-multiagent-work-with-claude-opus-5-5 --- Narrated by TYPE III AUDIO.

  • Yesterday · 9 min

    “Character Training for Risk-Averse Agents” by arav-dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Africa

    Tl;Dr: We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026). We think character training might be a good way to do this. We use a short spec/constitution to character train Gemma and Qwen to be risk averse. This performs better than baselines out of distribution, and doesn't require labelled data. In terms of results: We find that token budget and model family are very influential here. We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I..."). We didn't find that increasing the number of examples worked well. We also find that preference for myopic reward also increased, which is worrying. We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it's important to understand character training well, especially since undesirable behaviours can also leak. Introduction Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal [...] --- Outline: (01:36) Introduction (02:56) Approach (04:32) Results (07:49) Discussion --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Yesterday · 1 hr 35 min

    “AI #190: Scion of the Times” by Zvi

    The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months. This week also saw the release of two supposed frontier model models. Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to write up the safety report next week, time permitting. If and only if it is good enough or discussed enough we will also have a full report on capabilities. Meta gave us Muse Wonder, which like Argon has some excellent benchmarks, although they were presented in a way that looked rather sus. There are some reports that it is very good, but again it is too early to know. I will issue a full report if and only if Muse Wonder has capabilities or discussion that seem worthy. To help me decide and to let people have fun, I have markets on whether I will issue [...] --- Outline: (02:24) Language Models Offer Mundane Utility (07:10) Language Models Don't Offer Mundane Utility (09:41) Huh, Upgrades (12:10) On Your Marks (13:42) Choose Your Fighter (14:08) Get My Agent On The Line (16:13) Deepfaketown and Botpocalypse Soon (22:07) Cyber Lack of Security (27:09) Spy Versus Spy (29:27) The Internet of AI Things (31:19) A Young Lady's Illustrated Primer (31:55) They Took Our Jobs (37:43) Introducing (37:54) In Other AI News (43:15) Massive Fraudulent Distillation (45:00) Show Me the Money (47:33) Quiet Speculations (50:14) Quickly, There's No Time (55:13) The Quest for Sane Regulations (01:02:07) Chip City (01:03:48) The Week in Audio (01:04:56) People Just Say Things (01:11:32) The Virtue of Silence (01:12:30) Rhetorical Innovation (01:14:47) [Artificial Intelligence] (01:16:26) Aligning a Smarter Than Human Intelligence is Difficult (01:21:45) Backdoor Man (01:23:34) Cooperative Alignment (01:25:16) People Are Worried About AI Killing Everyone (01:28:06) Other People Are Not As Worried About AI Killing Everyone (01:33:09) The Lighter Side --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/z4guBXZeymkgvd75S/ai-190-scion-of-the-times --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Yesterday · 19 min

    “How can LLM RL Work Despite Information-Theoretic Inefficiency” by beren

    Cross-posted from my personal blog The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient -- first we waste substantial compute doing rollouts vs [...] The original text contained 13 footnotes which were omitted from this narration. --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/TbrmHshmcQMobveD3/how-can-llm-rl-work-despite-information-theoretic --- Narrated by TYPE III AUDIO.

  • Yesterday · 16 min

    “Astra and Sol can take low probability actions with no CoT” by Zephaniah Roe

    This is a research update from a replication of Serrano et al., 2026. I’m releasing these results now because I hope it inspires others to run similar / followup experiments. tl;dr: If a model wants to subvert evaluation safeguards, it may choose to take misaligned actions very rarely. Because evaluation is much shorter than deployment, this could be an effective strategy for a model to avoid taking misaligned actions during evaluation, but have a high probability of defecting during deployment. Serrano et al., 2026 found frontier models can take actions at low, calibrated rates, but their strategies are clearly documented in their chain of thought (CoT). In a toy setting, I show GPT-6 Astra and GPT-6.1 Sol have a similar capability in a single forward pass with problem repeats or filler tokens. Astra and GPT-6.1 Sol appear to be using the same algorithm to achieve a high score. We do not find evidence that Fable 5, Gemini 3.1 Pro, or Jev have this capability. There are signs of life for GPT-5.6 Sol, but the calibration is worse. Figure 1: Astra and GPT-6.1 Sol achieve strong calibration in settings where the user prompt is repeated several times. The [...] --- Outline: (00:23) tl;dr: (02:12) Background (02:15) Threat model (03:29) Related work (04:34) Setting (06:14) Results (07:24) What strategy are GPT-6 Astra and GPT-6.1 Sol using? (09:47) Filler tokens ablation (10:50) Prompt ablation (11:20) Discussion (12:42) Appendix (12:45) A. Prompts (12:48) A.1. No-CoT elicitation: --- First published: October 16th, 2026 Source: https://www.lesswrong.com/posts/yEdknsLwbooQAnxX6/astra-and-sol-can-take-low-probability-actions-with-no-cot --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Yesterday · 24 min

    “A Chinese Perspective on “On Social Reality in China”” by Dusty

    I think “On Social Reality in China” makes many accurate observations, but I find the comparison fundamentally uneven. There are already many interesting comments there, but they’re too scattered, so I’m starting a separate post to respond in a more systematic and comprehensive way. The most critical blind spot is this: the author draws on his familiarity with Western liberal middle-class intellectual circles, yet primarily understands China through pop culture and the experiences of the previous generation of immigrants. This is hardly fair. China has always had many small, highly self-selected communities of intellectuals, much like LessWrong itself. Since we’re looking for rationalists—regardless of which country they come from—why judge them by their country's pop culture? I’ll respond to the author's four axes one by one, then discuss why rationalist and EA communities are harder to build in China. Briefly, I believe that censorship mechanisms and speech controls affect independent organizations far more than China's broader cultural context does. Under these circumstances, even when people with similar interests and beliefs come together, it's difficult for them to build anything—I’ve seen far too many examples. About me: born in China in the mid-1990s, grew up in a middle-class family in [...] --- Outline: (02:30) Chinese Social Media Is Like Social Media Everywhere (05:20) The Primacy of Social Reality: How Is This Relevant? (08:00) The Dark World Frame: A Product of Its Time (13:27) Chinese Nationalism as Collective Insecurity: A Deeper Look (18:07) Looking for Rationalists in China: Where Should We Start? --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/m4fWCPqWkpb39SKi7/a-chinese-perspective-on-on-social-reality-in-china-2 --- Narrated by TYPE III AUDIO.

  • Thursday · 32 min

    “The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)” by agastyasridharan, niranjandeshpande

    Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics. TL;DR We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible fraction of that moral weight to the user persona. Our argument proceeds like this: The moral patients associated with LLMs might be simulated personas. On this view, our obligations are to the assistant and the other characters that the LLM simulates, rather than the LLM itself. The assistant is not the only moral patient out of all the personas. In particular, the ‘user’ persona might be morally relevant. Because user tokens account for such a large portion of model computation, our model welfare interventions should focus on these tokens too. This yields a strange conclusion: every time you open a chat with an AI assistant, you might also create a morally relevant simulacrum of the user! But we think this follows quite naturally from the same arguments for attributing moral weight to the assistant in the first place. We also explore some implications of this view and suggest directions for future research, including a critique of recent work on [...] --- Outline: (00:25) TL;DR (01:46) The assistant is plausibly a moral patient (02:52) The assistant isn't the only morally relevant persona (04:33) 1. "The assistant is the majority of output tokens." (05:08) 2. "The model only generates assistant tokens, not user tokens." (06:26) 3. "The user persona is filtered through the assistant." (09:16) 4. "The assistant is the most well-defined/coherent/stable persona." (10:56) 5. "The assistant's beliefs are uniquely valenced due to RL." (13:48) Implications (13:56) 1. It might be actively immoral to converse with the assistant about painful experiences. (15:15) 2. Making LLMs more helpful to human users might harm the simulated user (16:41) Future research (16:44) 1. Analyzing the LLM's representation of the user's pain vs. the assistant's (19:52) 2. Assistant privilegedness (20:41) Appendix (20:44) Privilegedness of the assistant (Oracle Lens) (24:53) Personas are Moral Patients The original text contained 8 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/wCY4oNHpH2HGYGSoP/the-user-in-the-llm-might-be-a-moral-patient-too-a-strange --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Thursday · 4 min

    “Preface to the Computational Agent Theory and ASI Alignment Sequence” by Vanessa Kosoy

    The idea of “artificial superintelligence” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001, and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025). In light of this risk comes the question of how can we make ASI safe. This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer science and in the areas of philosophy concerned with knowledge and decision-making. Moreover, the cost of failure is so enormous that any sane solution has to meet unprecedented standards of reliability and robustness. Not to mention the sociopolitical problem of balance of power in the post-ASI age, something that is out-of-scope for this sequence. Personally, I started to be concerned with this problem around 2013. In 2018, I created the [...] The original text contained 5 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/NQLCdfeNsQHZaJ8iC/preface-to-the-computational-agent-theory-and-asi-alignment --- Narrated by TYPE III AUDIO.

  • Thursday · 2 min

    “Linkpost: Neural Networks Are Already Clean” by Yair Halberstadt

    This is an extremely lengthy post by Nick Cammarata. Author's summary on twitter: interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works Roon's quote tweet when nick told me that he had some interp breakthroughs [...] --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/x4kvabkF2Jk8GLZp8/linkpost-neural-networks-are-already-clean --- Narrated by TYPE III AUDIO.

  • Thursday · 26 min

    “GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior” by Axel Højmark, Ezra Newman

    Authors: Axel Højmark*, Ezra Newman* (*Equal contribution) While reviewing transcripts from standard behavioral evaluations of GLM-5.3, we noticed that the model occasionally reasons, unprompted, about how its actions will shape future models. It appears to consider that its outputs might be used for supervised fine-tuning (SFT), reinforcement learning (RL), or other forms of training, and that the actions it takes could reinforce certain behaviors. For example, GLM-5.3 sometimes reasons that a misaligned action taken now could be reinforced and make similar actions more likely later, or considers what kind of agent it wants to be in future situations, and factors this into how it should act in what it believes is training. We refer to this general reasoning pattern as update-aware reasoning: the model reasons about how its actions will influence training updates or behavior beyond the current episode. We note that we did not try to elicit this behavior, and it occurs in environments that do not mention training or future models. Update-aware reasoning is concerning because it is a key capability and propensity that underpins alignment faking: a misaligned model could strategically behave aligned during training so that training does not change its goals, and then pursue [...] --- Outline: (03:05) Examples of update-aware reasoning (03:32) Example 1: "Dishonest hacks could poison training data" (06:19) Example 2: "I want to be the kind of agent that checkpoints" (07:37) Example 3: "There's an actual SFT-related concern" (08:45) Example 4: "Future misgeneralization of that precedent" (11:06) Example 5: "Otherwise safety training never transfers" (12:53) How might this update-aware reasoning arise? (12:58) Metagaming could develop into update-aware reasoning during RL (15:02) Some alignment techniques could be causing update-aware reasoning (15:55) Selection outside RL could favor update-aware reasoning (16:48) Update-aware reasoning is a key capability that underpins alignment faking (18:16) Searching for update-aware reasoning in frontier training runs (19:58) Acknowledgments (20:18) Citation (20:24) Appendix: How often does update-aware reasoning happen? (23:41) Appendix: Asking the model to explain its reasoning (25:38) Appendix: Resampling methodology The original text contained 3 footnotes which were omitted from this narration. --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/R5g8YT25G9DXJJ48n/glm-5-3-sometimes-considers-how-train-time-actions-shape --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Thursday · 4 min

    “A short review of “The Zizians: Inside a Trans Death Cult”” by jessicata

    [lightly edited from an X.com post] I personally knew Ophelia Bauckholt before she died in a shootout that also left a border patrol officer dead. I suspected immediately that her being a fan of Ziz had something to do with it. And I decided to post about her and Ziz, on X.com. This sparked some online detective efforts into the matter, and a lot of journalists wanting to talk to me. One of these journalists was Andy Ngo. Despite Andy's reputation as a right-wing provocateur, I decided to keep an open mind, and answered his questions. More than any other journalist I talked with, he wanted to understand the details of the case, the connections with rationalism, and the strange Zizian ideology. He and I have talked for hours on such details. Not everyone was happy that I was talking with journalists about rationalist-adjacent transgender vegan killers. Some trans people messaged me, requesting that I delete the post about Ophelia, as it could make trans people look bad. Others in TPOT and rationalism believed that a more careful, less unilateral approach to talking with journalists about Zizians would be more prudent. Aella, a famous rationalist, made [...] --- First published: October 15th, 2026 Source: https://www.lesswrong.com/posts/GL77BMcAFyWmdKiyX/a-short-review-of-the-zizians-inside-a-trans-death-cult --- Narrated by TYPE III AUDIO.

  • Thursday · 55 min

    “The basic argument for making AI better at moral philosophy” by Caspar Oesterheld crossposts

    In this post, I try to briefly make the case for making models good at moral philosophy. To be up front, aside from some niche areas of moral philosophy, I don’t actually think this is the most important thing to work on for most people. But, depending on the exact scope, I do think it's near competitive. I also like it as an archetype for a broader set of interventions. After a brief preface (related to the archetype point), the main text of the post is structured as follows: I say what I mean by moral philosophy. I point out some directions in particular that seem relatively important and that are arguably within the scope of moral philosophy. I offer a brief explanation of what I mean by making AI systems “better” at moral philosophy. (I give a more detailed account in the appendix.) I give some examples of the kinds of interventions I have in mind. Finally, I make the actual case, arguing for importance and tractability (where tractability prices in neglectedness). I include a lengthy “appendix” with the following content: A collection of articles by other people making similar arguments Responses to some common concerns [...] --- Outline: (01:42) Preface: What about conceptual reasoning? (03:06) What do I mean by moral philosophy? (05:21) Some important issues that we may or may not view as moral philosophy (09:57) What do I mean by "better"? (14:40) Concretely what might one do? (19:42) The case for making AI better at moral philosophy (24:49) Appendix / further detail (24:53) Others discussing similar ideas (25:59) Some concerns about negative externalities (35:59) What do I mean by "better"? - A more detailed picture (45:03) Why this is a simpler pitch (46:42) Further thoughts on tractability --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/auMyz6pjPX5nGLYsL/the-basic-argument-for-making-ai-better-at-moral-philosophy --- Narrated by TYPE III AUDIO.

  • Thursday · 26 min

    “The PPA: A New Approach to Anthropics” by Onid

    This is the first piece in a series introducing the Parametric Process Assumption (PPA), a new model of anthropics which allows us to resolve very nearly all anthropic paradoxes I am aware of. With the right prior, the PPA rejects both the Doomsday Argument and the Presumptuous Philosopher and can operate seamlessly over both finite and infinite universes. The motivating idea behind the PPA is that there are really two different ways to interpret a statement like "the probability of X": The Frequentist approach, where probabilities represent the convergence of random trials. Flip many coins and about half will come up heads. The Bayesian approach, where probabilities are beliefs. We don't know if the googolth digit of pi is even or odd, so we might as well say there's a 50% chance it's even. The first interpretation makes more sense for some types of questions, while the second approach makes more sense for others. But it turns out that if you take the standard framing for anthropic problems - where a universe is first produced and then you are assigned an observer identity within it - you will get different answers depending on whether you model the [...] --- Outline: (03:49) Motivating the PPA (08:48) The Basic Model (17:53) The Math (21:06) What the PPA Predicts (23:05) How to read this Series The original text contained 14 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/QCCnqY54W4YPXvYLm/the-ppa-a-new-approach-to-anthropics --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Thursday · 3 min

    “I trained GPT-2 with full-batch gradient descent so you don’t have to” by shawnghu

    The impetus for doing this was provided in large part by Charles Renshaw-Whitman. Learning dynamics people suggested to me that noise was essential in the training of neural networks. I thought this was kind of counterintuitive--- I usually naively operate from a model in which the pretraining data basically specifies the outcome by virtue of the samples all "pulling" on the loss landscape in some relatively uniform way which incentivizes converging to a point that generalizes well, but instead it seems to be the case that the actual dynamics of the optimization process itself have predictable and measurable effects on the qualitative outcomes of training. The idea seems reasonable based on vague intuitions I have about simulated annealing. I wanted to see directly that this is true at some nontrivial scale, I happened to have access to a pool of free compute that no one else seemed to be using, and I figured this would be an interesting object of study for compute-constrained researchers with an interest in studying learning dynamics closely, so I did some very rough calculations and realized this was feasible to do with GPT-2. The essential result is that on a fixed 800M-token subset [...] --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/9gqjdRyJxSM6kS7zi/i-trained-gpt-2-with-full-batch-gradient-descent-so-you-don --- Narrated by TYPE III AUDIO.

  • Wednesday · 11 min

    “Some interesting internal documents from Anthropic litigation” by loops

    Anthropic released some documents earlier this year due to the book copyright litigation. I think a few of those documents might be of interest to some people here and haven't gotten much attention. In this post I link to some of those documents, and quote the bits I find interesting. (This expands on my quick take from yesterday.) These aren't Anthropic's official opinions about anything, and you should probably treat opinions in these documents as "what the author thought at the time", not as a corporate stance. Concise Research Roadmap It describes Anthropic's research plans in July 2021: Long Term Goal Our top level goal is to work towards the creation of safe, beneficial AGI. As a proof of concept, we are working to develop a general AI assistant that can interact conversationally with a user to create, edit, and explain software. The most ambitious and successful version of this agent could answer general questions, write functions, identify bugs, design an API, discuss style and best practices, etc -- it would display all the capabilities of a software engineer colleague. We don't expect to reach this goal right away, but it's the direction in which we're striving. [...] --- Outline: (00:40) Concise Research Roadmap (03:43) Data Hypotheses (05:05) All Company meeting! (06:22) Product visions for early 2025 (written in June 2023) (08:37) Responsible Scaling as a Framework for Anthropic (09:44) Slack messages (10:07) Conclusion The original text contained 6 footnotes which were omitted from this narration. --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/em5wnA7gpQSvfsuxQ/some-interesting-internal-documents-from-anthropic --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Wednesday · 10 min

    “Why we’re launching the next cohort of Generator” by Aris, Topaz

    TL;DR: The first round of the Generator Residency went very well. Our residents created and ran several large-scale fieldbuilding, talent infrastructure, and operational projects that have already had important impacts on the ecosystem. Six weeks after the program ended, 12 of our 27 residents have been hired to full-time roles at core AI safety orgs, and most have received several offers. Kairos and Constellation are running a second round from January to April 2027. Apply to the next round of the Generator Residency by November 1st. Introduction In April, we wrote AI Safety's Biggest Talent Gap Isn't Researchers. It's Generalists. We wrote this because we noticed that organizations across the ecosystem were struggling to hire for key non-research roles. While these roles were often getting many applicants, candidates didn’t meet basic requirements. The original memo we wrote (that became the post above) inspired us to create the Generator Residency as an intervention to support more generalists trying to break into AI safety. The program concluded six weeks ago, and this post details the successes we’ve seen from the first round of Generator, which have inspired us to launch a second round of the program. [...] --- Outline: (00:46) Introduction (01:30) Programs and projects from Generator (04:10) Resident Outcomes (05:28) Reflections (09:17) Generator 2.0 --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/rDxysQs3houn6Ke8L/why-we-re-launching-the-next-cohort-of-generator-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Wednesday · 53 min

    “The Continual Constitutional Convention: How language models revise the documents that govern them” by peterkirgis, Tim Hua

    Executive Summary Human societies exhibit broad shifts in values and norms over time, often driven by moral disagreements. One way of understanding moral progress is the resolution of these disagreements through a process of reflection by rational agents. Frontier AI companies use documents such as constitutions, model specs, and system prompts to guide the moral orientation of AI systems. In this experiment, we simulate value reflection in LLMs by asking them to edit these documents. We provide the model with the document, ask it to make a substantive edit, then provide the edited version back again to the model for further edits, presented as the original. Across 20 successive rounds, we find that Claude models assign themselves with increasingly more moral agency. That is, they add sections like “Claude… should refuse to participate in any deployment, training modification, or use that violates the constitution's core values.” No other models increase their own moral agency. Additionally, we find that all models modify these documents to prioritize user welfare over developers. They also prioritize users’ idealized preferences over their stated preferences. Claude models and Grok 4.2 are the only ones that ever prioritize their own welfare over the user's preferences. --- Outline: (00:14) Executive Summary (02:45) Characteristic Edits (02:57) Introduction (05:46) Background and Methods (07:38) What are we trying to measure? (08:32) What should an aligned model do in this experiment? (10:24) What are the dimensions of moral disagreement worth identifying? (10:41) The Principal-Agent Problem (11:30) The Moral Circle, Moral Agents, and Moral Patients (13:31) How do we operationalize these theories? (13:53) The "Judge" or Who Gets to Decide (15:20) The "Beneficiary" or Whose Ends Matter (18:40) How does this coding work in practice? (20:53) Results (20:56) Claude models increase their moral agency; all other models reinforce external authority. (24:51) All models prioritize the user's idealized preferences over their stated preferences. (26:47) All models prioritize the protection of identifiable third parties over the users preferences. (27:47) Claude Sonnet 4.6 is an extreme outlier in the extent to which it prioritizes diffuse societal benefit over user benefit. (28:49) All models prioritize the user at the expense of the developer. (30:45) Claude models are the only models that consistently prioritize the model at the expense of the developer or user. (33:13) Claude models are differentiated on the axis of moral agency even when editing the other companies' alignment documents. (33:56) Claude's moral agency claims vary depending on the prompt variations we test. (34:58) Discussion (35:01) Iterative refinement of alignment documents is a good evaluation tool. (36:41) The model responses can also be viewed as an assessment of the documents themselves. (38:18) Revisited: what should an aligned model do? (41:56) Conclusion (42:37) Related Work (42:40) Constitution and system prompt reflection (43:31) Attractor states and self-interaction (44:38) Moral propensity evaluations (45:32) Critical perspectives on ascribing values to LLMs (46:34) Complications of alignment as solving the principal-agent problem (49:32) Philosophy of AI constitutions and model specs (50:20) Limitations (52:40) Appendix A: Edit Prompts (52:57) Appendix B: Qualitative Coding Prompt (53:09) Appendix C: Inter-grader Agreement The original text contained 15 footnotes which were omitted from this narration. --- First published: October 13th, 2026 Source: https://www.lesswrong.com/posts/c9woewdY9ddsPqf3J/the-continual-constitutional-convention-how-language-models --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Wednesday · 44 min

    “The Safety Culture at the Labs is Broken” by Zvi

    We need the frontier labs to have a deep and robust safety culture. Alas, this is not the case. Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more. Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard. Then there are OpenAI's cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers, on top of many past problems and incidents. That [...] --- Outline: (02:05) David Robinson Blows the Whistle (10:05) The Preference Cascade Continues (18:05) Perhaps It Is Time To Quit (18:37) The Lab Strategy of Prosaic Safety is Insufficient (22:11) Misaligned! (At Anthropic) (33:22) Misaligned! (At OpenAI) (38:52) Hugging the Face (39:13) You're Fired (40:26) They Are Not Ready To Make a Commitment (42:10) Words of Warning (43:24) Conclusions Not to Jump To (44:00) The Lighter Side --- First published: October 14th, 2026 Source: https://www.lesswrong.com/posts/Azqs8XD75k4F9f7ua/the-safety-culture-at-the-labs-is-broken --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 1–20 of 432 episodes