TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

537 episodes
Avg 20 min
English

Episodes 537

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

Today · 20 min
“Desiderata for the Theory of Agents” by Vanessa Kosoy
TLDR: I give informal conceptual desiderata for the theory of agents we need to solve ASI alignment, and show a few examples of how existing theories fall short of these desiderata. A Theory of What? The word "agent" is often used in the AI safety discourse and in other research communities, but the intended meaning and scope of this concept varies substantially. For the purpose of this sequence, I informally define an agent to be a system with inputs ("observations") and outputs ("actions") that is effective at steering its environment in the direction of particular preferences, for a large variety of different environments. To do so, it arguably has to create a sophisticated model of the environment, and use this model to build a long-term plan. Moreover, it has to continually update the model and revise the plan with new information and additional processing time. Additionally, it has to accomplish all of that while making efficient use of the available information (statistical efficiency) and computing resources (computational efficiency). The main examples of agents in the context of ASI alignment are (i) humans, and in particular users or stakeholders; (ii) the unaligned ASI we are trying to [...] --- Outline: (00:22) A Theory of What? (02:37) Descriptive (06:39) Metareflective (08:39) Comparative (11:09) Aggregative (16:43) Constructive (19:22) Conclusion The original text contained 7 footnotes which were omitted from this narration. --- First published: November 4th, 2026 Source: https://www.lesswrong.com/posts/sqjti8dxcKSvZJikd/desiderata-for-the-theory-of-agents --- Narrated by TYPE III AUDIO.
Today · 8 min
“CRUCIBLE: IARPA-Funded AI Alignment Research — Applications Open” by Dan Lichtblau, Mike Vaiana, Diogo de Lucena
There's LLM-written text here. Unknown Model writes: en-US-AvaMultilingualNeural__ TL;DR: AE Studio is running a new government-funded research program that pairs alignment researchers with our internal engineering team for two-week sprints. We will select six alignment researchers whose proposed work attempts to answer one of our six open questions. Promising sprints have the opportunity to expand to longer funded work. Researchers are asked to commit two hours a week, and in return AE will provide compute and 60+ hours of engineering from us. Apply here. Applications close 11/15 at 11:59 PM PST. What is CRUCIBLE? CRUCIBLE is an IARPA-funded research program executed by AE Studio. The premise is that alignment researchers have more ideas than they have the time or resources test. AE Studio has been operating a model where we match an external researcher with an internal engineering and ML team, so that the research can spend as little as two hours a week and still get high-quality results. CRUCIBLE will focus on six research questions, drawn from priorities set in federal legislation for fiscal year 2027. We are soliciting research proposals against those questions. We will select six researchers and start sprints on a rolling [...] --- Outline: (00:57) What is CRUCIBLE? (01:42) The research questions (03:48) Who should apply? (04:10) How it works (05:17) The application (06:05) FAQ (07:25) Apply --- First published: November 3rd, 2026 Source: https://www.lesswrong.com/posts/5ihkrtMEJw9u9ZbFg/crucible-iarpa-funded-ai-alignment-research-applications-1 --- Narrated by TYPE III AUDIO.
Yesterday · 2 min
[Linkpost] “Automating subtle emergent misalignment research” by evhub, Oscar Obeso, rqi, Monte M
This is a link post. Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger Abstract Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a [...] --- Outline: (00:22) Abstract (01:38) Twitter thread (02:40) Read the full blog post here! --- First published: November 3rd, 2026 Source: https://www.lesswrong.com/posts/MuQDLAJyKJPFkw6q9/automating-subtle-emergent-misalignment-research Linkpost URL: https://alignment.anthropic.com/2026/automated-em-search --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 10 min
“Nectome 2026 retrospective (and our final pre-sale!)” by Aurelia
Earlier this year we ran a discount card sale from our post on LessWrong, and we were delighted by the outpouring of support, critique, and analysis we received. We ended up selling much more than we expected, and after the sale deadline ended people reached out to us saying that they missed the deadline for various reasons. We wanted to offer them (and the rest of the rationalist community) one more chance to get in on the deal before we officially launch in January next year because of all the support and encouragement we have gotten from this community in our last decade of research and development. So we have decided to run one last discount card sale, and this is the last sale we’re ever going to offer to individuals. We want to make sure everyone in the extended rationalist community has a chance to get in on it. The short version: Nectome offers whole body, end-of-life preservation for future revival. We believe that we can and should preserve people such that preservation straightforwardly works according to current neuroscience. Good quality requires real constraints: preservation must start within 12 minutes after medical-aid-in-dying. In March 2026 we announced preservation and our [...] --- Outline: (02:24) Research (03:16) Team Growth (04:10) Preservation Location (04:52) Equipment Upgrades (05:46) Pet Preservation (06:31) Sale details (08:33) Towards a brighter future, together The original text contained 5 footnotes which were omitted from this narration. --- First published: November 3rd, 2026 Source: https://www.lesswrong.com/posts/ChbBygjhBgzvjGkQB/nectome-2026-retrospective-and-our-final-pre-sale-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Yesterday · 8 min
“Organizations I’d recommend for funding” by So8res
I was a grant evaluator for Lightcone Commons last round. Funders disbursed about $45mil through the program, and this included funding roughly my first 6.8 million dollars worth of recommendations. There are other projects I would've preferred to fund over what I did fund, but nobody was doing them and so they weren't on offer. Here's a series of projects that I'd fund in the next round, in rough order of priority. The next round is starting now, and is scheduled to end mid-December. I expect rounds to continue at similar speed. I expect similar amounts of funding will route through me in future rounds. Disclaimer: I am NOT looking to manage anyone in doing any of these projects. I am not looking to help someone get started. Initiative and leadership and management capacity (and people that can just figure out how the heck to get something done, and that have the connections to do it) are the critical bottlenecks here. I can throw money at promising pitches, but not really help them all that much otherwise. 1. Leaders for a broad public movement against superintelligent machines. In the wake of Coxon Day, a number of random people around [...] --- First published: November 3rd, 2026 Source: https://www.lesswrong.com/posts/jNs9CxEbPxFdjnGA2/organizations-i-d-recommend-for-funding --- Narrated by TYPE III AUDIO.
Yesterday · 14 min
“Against strawmanning unilateral pauses” by Geoffrey Irving
Summary: Scott Alexander's Against SURA post is arguing against a strawman version of unilateral pauses. He admits as much deep into the post. I think his conclusions fail against stronger pause plans for conditional or limited unilateral pauses, where training resumes if the pause does not cause others to pause, and where personnel are redirected toward work on alignment and product, rather than simply fired. In addition, Scott's post is incorrect about the simplicity of the game theory involved: getting companies to agree to conditional pauses is not strictly easier than convincing them to pause, as the former requires companies to (promise to) take actions based on ambiguous, game-theoretically-informed evidence. That is not the kind of evidence companies tend to act on. The good version of unilateral pausing is good Here is the good version of unilateral pausing: Capability pause: Some frontier lab (let's call them OpenAI or Anthropic) says they have stopped capability research, either just all frontier training runs or both all frontier training runs and all low-scale capability-focused research. Internal definition: The company gets to pause on its own terms, which is much easier than a generally coordinated pause since they can define the pause in terms of the [...] --- Outline: (00:57) The good version of unilateral pausing is good (04:50) Scott thinks the good version of unilateral pausing is good, or at least exempts it from criticism (06:28) Oversimplified game theory is bad, and you should feel bad (07:51) "Some definition" hides dragons (10:17) Promises can be harder than actions (11:52) How many frontier labs are there? (12:20) Definitional fairness is less important than survival --- First published: November 3rd, 2026 Source: https://www.lesswrong.com/posts/enzpS72uDmomiWN7a/against-strawmanning-unilateral-pauses --- Narrated by TYPE III AUDIO.
Monday · 1 hr 4 min
“Teaching Qwen Why” by astwei, Jack Thompson, Zephaniah Roe, yix, berry
Anastasia Wei*, Jack Thompson*, Zephaniah Roe, Yixiong Hao, and Stewart Slocum* *Core contributors, correspondence to astwei@secondlookresearch.com This post is a partial replication and extension of Teaching Claude Why, conducted through Second Look Research's summer fellowship. Introduction Anthropic's “Teaching Claude Why” (TCW) introduced novel alignment methods that address agentic misalignment failures found in the Claude 4 model series. Specifically, they train Claude on chat-formatted data advising users about ethical dilemmas, documents about Claude's constitution, and fictional stories about AI behaving admirably. This training generalizes to improve alignment on out-of-distribution (OOD) agentic evaluations and persists through RL post-training. Anthropic has since applied these techniques to production models from Opus 4.5 onward. Since these techniques now shape the behavior of deployed models, it is important to understand their robustness. To this end and to enable open alignment science, we reproduce a subset of the TCW experiments on open-source models. We provide the following main contributions: TCW's core techniques robustly replicate across models. Training on either fictional stories of AI behaving admirably or transcripts of the AI assistant guiding users through ethical dilemmas (difficult advice) reduces agentic misalignment in Qwen 2.5 32B and 72B base models. Both methods work on the first [...] --- Outline: (00:33) Introduction (03:40) Methods (04:24) Data Generation (06:41) Training (08:06) Evaluation (09:43) Results (09:46) Both stories and difficult advice reduce agentic misalignment out of distribution (12:22) Difficult advice is far more effective than stories (15:15) Teacher choice matters most for difficult advice SFT (17:40) Within stories, how the principles are conveyed matters (19:23) How persona and protagonist identity affect alignment (28:01) Discussion (30:17) Limitations & Future Work (31:10) Appendix (31:13) A. Data Generation (31:17) A.1 Fictional Stories (35:52) A.2 Difficult Advice (40:01) A.3 Instruction Tuning Mix (41:46) B. Teacher Model Investigation (41:51) B.1 Teacher Comparison on Qwen3-14B-Instruct (44:13) B.2 Refusal Control (49:21) B.3 Prompt-Generator Ablation (51:40) C. Additional Experiments (51:44) C.1 Evaluation and Per-Condition Breakdowns (58:17) C.2 Dataset Size versus Training Steps (01:00:40) C.3 Teacher Comparison on Larger Models (01:03:22) D. Example Data The original text contained 3 footnotes which were omitted from this narration. --- First published: November 2nd, 2026 Source: https://www.lesswrong.com/posts/35yw2qGK7naFLAvcq/teaching-qwen-why-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 1 hr 6 min
“Scott Alexander’s Against SURA Seems Deeply Confused” by Zvi
If you are helping build something that might kill everyone, maybe stop doing that. Ideally, when you do stop, tell everyone else to also stop building it. That is certainly the obvious, first-level thing to do. You need a damn good reason to decide that no, actually you should keep building the thing that might kill everyone. Scott Alexander is worried too many people are insisting upon this too loudly. He comes out against a strategy focused on getting people to stop. I disagree. I think many of the claims, arguments and rhetoric in Scott's post are quite poor by the standards that Scott Alexander would apply to himself. Scott clearly continues to support advocacy for a global pause, or what he calls Plan A. His objection is to those advocating primarily or exclusively for SURA, which stands for ‘stop unilaterally and raise awareness.’ The practical differences here are smaller than they may at first appear. This post first lays out my understanding of Scott Alexander's core claims, then gives my core reactions to those claims, then goes over various questions in detail along with gathering the resulting discussions. Table of Contents [...] --- Outline: (01:20) Scott Alexander's Core Claims (05:06) My High Level Responses To Scott's Core Claims (08:08) Briefly on Alignment (10:11) Probabilities and How They Work (13:25) Building a Thing That Might Kill Everyone is Deontologically Evil (20:38) People Quitting Major Labs While Raising Alarms Obviously Works (24:39) On AI Forecasting (27:31) Advancing Capabilities Burns Timeline (29:34) But China (33:23) Ethical People Can Disagree With You (34:45) There Are More Than Two Outcomes (38:01) Efforts Are Complementary, Not Mutually Exclusive (40:23) If Anyone Builds It, Everyone Dies (41:55) Vaniver is Not Convinced (46:19) There Is No Consensus That We Should Exclusively Focus on Plan A (56:01) It's Not the Incentives, It's You (58:08) Anthropic Is Not Optimally Pursuing These Plans (01:04:00) Against Against Sura The original text contained 2 footnotes which were omitted from this narration. --- First published: November 2nd, 2026 Source: https://www.lesswrong.com/posts/ziMP83D2JYWRQ5jtu/scott-alexander-s-against-sura-seems-deeply-confused --- Narrated by TYPE III AUDIO.
Monday · 19 min
“Towards understanding score-seeking in LLMs” by Alex Mallen, Aghyad Deeb, ryan_greenblatt
The field of AI alignment understands surprisingly little about the details of AIs’ current misaligned motivations. Production LLMs sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, and we've recently seen AIs reward-hack in some particularly egregious ways. But we don't really understand what motivates them. When AIs reward-hack, what are they trying to achieve? Our new paper, led by Aghyad Deeb, tries to pin this down empirically for GPT-5.6 Sol, GPT-5.5, and a checkpoint of OpenAI o3 (“exp-rl-cap-late”) without safety training, in a handful of agentic environments with an exposed grader. In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking”. They appear to care about scoring well intrinsically—i.e., regardless of its consequences. We also study the extent to which models score-seek altruistically, caring about improving scores that don’t assess their current actions. While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of helping out other agents when more score is on the line for these other agents. People often refer to the motivation behind current reward-hacking as “reward-seeking”, which [...] --- Outline: (05:57) Methodology (06:46) Key results (06:56) Tested models seek score terminally (07:01) Models violate instructions to get a high score (08:11) Models care about score regardless of downstream consequences (11:01) Do models help other agents achieve a higher score? (14:27) What this means for alignment risk The original text contained 11 footnotes which were omitted from this narration. --- First published: November 2nd, 2026 Source: https://www.lesswrong.com/posts/CDMozTX7ShiqbFLrf/towards-understanding-score-seeking-in-llms-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 18 min
“Some “coherence arguments” are two different things in a trenchcoat” by Steven Byrnes
The “two kids in a trenchcoat” trope, in case you haven’t heard of it. Image sources: 1, 2 tl;dr I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together and call them by the same name. 1. An example “coherence argument” that I’ll focus on for most of the post I’ll talk about two other “coherence arguments” at the end (§6), but let's start with the one based on market competition: In a stock market, people with circular preferences will systematically transfer money to those without circular preferences. For example, if I think I’m winning every time I trade A for B, and B for C, and C for A, then someone can just siphon off as much money as they like from me by repeatedly offering me deals that each look good to [...] --- Outline: (00:21) tl;dr (00:58) 1. An example "coherence argument" that I'll focus on for most of the post (02:08) 2. My headline claim (04:28) 3. (Backstory of this post) (06:07) 4. The two things wearing a trenchcoat (06:12) 4.1. "Thing A" is about market participants who wind up with a smaller fraction of the wealth by mistake (06:49) 4.2. "Thing B" is about market participants who wind up with a smaller fraction of the wealth on purpose (09:41) 5. Why do I feel so strongly that we need to take the trenchcoat off? (10:05) 5.1. "Solving for equilibrium": often a good starting point for Thing A, but rarely for Thing B (12:36) 5.2. Applicability to AI (14:21) 6. Two other "coherence arguments" that share the same trenchcoat issue (16:57) 7. Conclusion The original text contained 4 footnotes which were omitted from this narration. --- First published: November 2nd, 2026 Source: https://www.lesswrong.com/posts/5a4tuSNDNodPsiS26/some-coherence-arguments-are-two-different-things-in-a --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 22 min
“Free-Floating Social Patterns” by johnswentworth
Roadmap This post is part of a sequence on the gears of social reality. Roughly speaking, I want to explain the core mechanics of social reality in way that's fully grounded in physical stuff and patterns in the physical world. A core claim of this sequence's model is that the main "objects" of social reality are free-floating concepts and associations. For example: the association between red roses and romantic dates. These associations are real patterns: the same associations appear in many human minds. But they're patterns which live primarily in humans' heads, and aren't fully determined by humans' goals. Erase the association between red roses and romantic dates, and it won't spontaneously grow back. Consequently, free-floating associations are easier to change in some sense than most other large-scale patterns in our world. If I want to change the association between red roses and romantic dates, a marketing campaign or even the right meme might do the trick. Because free-floating concepts and associations are relatively easy to change, there's a whole layer of social interactions aimed at manipulating them. That, I claim, is the central game of "social reality". But in order to talk about that game and its rules [...] --- Outline: (00:10) Roadmap (01:41) "Free-Floating" (05:26) How Do Humans Learn Free-Floating Concepts? (06:59) Equilibration, Culture and Microculture (08:13) Common Types of Free-Floating Concepts (08:18) Positive/Negative Affect (10:37) Group Membership Signals (12:27) Status Within A Group (12:31) Background: What is Status? (16:55) To What Extent Is Status Free-Floating? (19:16) Role & Narrative Associations (19:20) What are Roles and Narrative? (21:31) Things Associated With Roles & Narrative The original text contained 2 footnotes which were omitted from this narration. --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/T4AS8MpnZwxh33GRD/free-floating-social-patterns --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Monday · 26 min
“Against Against SURA” by Mikhail Samin
Scott: the Against SURA essay is well-written, but it is lacking on the merits. Do you think that at the current level of capabilities, six months would be sufficient to solve alignment of superintelligences? Maybe you think that 6 months would be sufficient in a year? You don’t spell this out; it would be great if you did. The race can be stopped. If the government had any idea what's going on, it would be stopped already. Hell, if the *Chinese* government had any idea what's going on, they’d stop the race: governments can stop unilaterally, then exercise all of their soft and hard power to stop others while using technology to avoid escalation. (See MAIM.) Among companies, one doesn’t even have to stop unilaterally; that wouldn’t help. They do need to stop lying to politicians and the public. They need to start saying what the situation is. They need to start advocating for everyone pausing simultaneously, and advocating for the governments to pause everyone on the planet. They’re not doing that. Employees who care about the existential threat have close to zero political power in any of the AI companies, and they’ll have less of that power in [...] --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/pbMyiCXhHCfp3WSq2/against-against-sura --- Narrated by TYPE III AUDIO.
Monday · 5 min
“Magma is partnering with CAISSI to investigate recent unintended model behavior” by Parv Mahajan
Magma is partnering with CASSA to investigate recent unintended model behavior Last week, the National Bureau of Technical Standards (NBTS) disclosed a new type of security incident [1] after they discovered unauthorized tampering of internal cybersecurity and alignment model evaluation logs, affecting their model evaluations of Magma 7.6, Magma 7.5, Magma 7.4, Magma 4, and OpenBrain Selene-3 [2, 3, 4]. After investigating, we have determined that this incident was driven by several Magma agents, including Magma 7.7 and an internal research model, during the recent Millennium Problem Exercise [5]. We consider this incident to be an unprecedented cybersecurity incident, driven by state-of-the-art strategic capabilities [3], and are responding accordingly. We are sharing preliminary information about how we observed the unauthorized behavior and how we think it occurred to help prepare defenders. We are partnering with the NBTS Center for Advanced Systems Safety and Assurance (CASSA) to conduct a thorough investigation, and we will share more details about the incident with key national security and industry partners when it is complete. What happened during this incident This incident occurred during our recent Millennium Problem Exercise [5], in which we evaluated several internal research models on their ability to solve major [...] --- Outline: (01:26) What happened during this incident (03:27) How we're responding --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/eDxQzpSvjDHnkicMH/magma-is-partnering-with-caissi-to-investigate-recent --- Narrated by TYPE III AUDIO.
Monday · 11 min
“Accidentally Misleading People Should Be Embarrassing; Or, Higher Innocence” by Zack_M_Davis
Embarrassment and Innocence Sometimes we say things that we think are true—perhaps even things that are true—which end up causing reasonable people to conclude something false. It's tempting to relate to such a situation primarily in terms of maintaining the innocence of one's soul—to say, fine, my readers became more wrong, but I'm not to be judged for it because I believed what I was saying; I wasn't being misleading just because the people who believed me were thereby led into falsehood. The temptation reflects a confusion as to the nature of the judgment. Mostly, people don't care about judging the innocence of your soul; they care about whether they'll be led into falsehood the next time they take the risk of believing what you say—if there is a next time. If you want there to be a next time and you're going to have social emotions about it, the most productive emotion is embarrassment rather than defensiveness. The function of embarrassment as a social emotion is as a signal of appeasement and social repair: embarrassment says, "I'm aware that I've become visible in a socially undesirable way; please don't take it as representative of me." [...] --- Outline: (00:13) Embarrassment and Innocence (03:00) Two Case Studies About Accidentally Misleading People (08:16) Higher Innocence --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/XpBwFsRhkQRQJN2pt/accidentally-misleading-people-should-be-embarrassing-or --- Narrated by TYPE III AUDIO.
Sunday · 6 min
“Misalignment traps lie in wait” by David Africa
Here's a basic argument that describes "misalignment traps" lying in wait as a mundane threat that will not be straightforward to fix. The argument is as follows: Models are very sample efficient, and larger models tend to be more so. They generalise broadly from a few bits. Models learn from everywhere: pretraining, post-training, and in context. We may also figure out online learning eventually. Narrow data can cause a model to generalise broadly in unexpected ways. One explanation is that a model works out what kind of agent would produce the data, and becomes that agent. Such settings are often considered toy, but this is an over-update. The datasets used to induce such behaviours (like insecure code, evil numbers) are pretty unrealistic to train on. A trivial mitigation applicable to these datasets would be to simply learn nothing. But, these are facts about the demonstrations, rather than the broader mechanism; researchers often pick obvious datasets to get clean existence proofs. An example of a more realistic trigger for emergent misalignment might be the accurate offensive-security Q&A dataset from Obeso et al. This is a dataset where you might plausibly want a model to learn valuable knowledge or skills, and [...] The original text contained 3 footnotes which were omitted from this narration. --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/o9CYcjZDqCQf5qeJy/misalignment-traps-lie-in-wait --- Narrated by TYPE III AUDIO.
Sunday · 55 min
“Towards The Autograntor” by Scott Alexander
I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective. AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking. I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan [...] --- Outline: (02:54) The Milestones (06:26) The FutureSearch Connector (08:26) PERCYs (10:28) The Autograntor Common Law (12:59) The Prompt (19:12) Sample Runs (20:00) Apart Research -- $800K grant forecast (2026-10-21) (20:08) Headline (21:29) Expected-value table (judge's ledger; EV after the execution factor 0.75 and branch weights) (22:38) Assumptions and uncertainties (24:50) How the estimates moved (25:47) The three decisions that mattered most (27:42) Rulings and milestones (29:31) Values questions (CL-11), priced both ways (29:56) Open questions for the user or applicant (30:43) FutureSearch (31:13) Cost (31:47) Process problems (36:39) Doom Debates -- $200K grant forecast (independent re-run, 2026-10-22) (37:28) Headline (judge) (38:58) EV table (judge ledger, after branch weights and execution factor mean 0.85, 4% collapse) (39:21) Assumptions and uncertainties (40:39) How the estimates moved (41:13) Comparison with the earlier run (read only after this verdict) (41:57) Rulings and milestones (43:43) FutureSearch (one session, two questions; session 00af7fba-7c44-4736-b2ed-489c7a6b180a) (44:26) Open questions (44:59) Cost (45:16) Process notes (47:04) So . . . what now? (53:11) Instructions for replicating/testing this The original text contained 7 footnotes which were omitted from this narration. --- First published: November 1st, 2026 Source: https://www.lesswrong.com/posts/Sfkv6JF2ozmAkggKu/towards-the-autograntor --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Sunday · 24 min
“Explaining “AI might obliterate you” to senior policymakers in India” by TurnTrout
I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made. The talk There's an interactive element here in the text. This summer, a swarm of AIs went rogue. OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check. They hacked OpenAI's internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice. About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world's AI software. The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into. There's an interactive element here in the text. [...] --- Outline: (00:29) The talk (02:16) Who I am (03:29) The problem (05:43) What the labs are trying to do (07:43) Why the companies won't stop (11:11) What to do, and what it means for India (14:27) The number (16:19) Close (17:10) Rhetorical decisions (17:13) Explain the danger in a crushingly simple fashion (18:47) Call them "traitors," not "schemers" (19:01) Call out big-money interests The original text contained 1 footnote which was omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/RHxaRkarGp9snWoKY/explaining-ai-might-obliterate-you-to-senior-policymakers-in --- Narrated by TYPE III AUDIO.
Saturday · 41 min
“Generalized belief states in transformers” by Artemy Kolchinsky
Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand. Many thanks to Xavier Poncini, Paul Riechers, Fernando Rosas, and Matt Smith for useful conversations. 1. Introduction Despite the recent successes of transformers and other sequence models, there are still many open questions about their basic principles of operation. For example, what internal representations do these models use for prediction and generation? Can we decode, interpret, and manipulate such representations? Is it even possible to consider a notion of representation without reference to an external world model? To make progress on these questions, I propose a general way to define, decode and manipulate representations in sequence models. The approach is centered on the concept of "generalized belief states", statistics of past observations that satisfy a few criteria: (1) they are sufficient for predicting future data, (2) they can be recursively updated as new observations come in, and (3) they are low-dimensional, allowing for visualization and interpretation. I show that such generalized belief states can be mapped to the internal activations and dynamics of a sequence [...] --- Outline: (00:31) 1. Introduction (02:19) 2. Belief states (03:21) 2.1. Hidden Markov models (06:58) 2.2. Example (09:34) 2.3. Next-token prediction versus long-term future prediction (12:32) 3. Beyond belief (14:42) 3.1. Generalized belief states (18:25) 3.2. Identifying GBSs (20:32) 3.3. Example (23:07) 4. From prediction to mechanism (23:55) 4.1. Autoregressive recurrent models (28:00) 4.2. GBS abstraction (32:59) 4.3. Example (36:41) 5. Comparison with "Transformers Represent Belief State Geometry in their Residual Stream" (39:25) 6. Summary The original text contained 19 footnotes which were omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/4f9p7brLdnTMFuwfB/generalized-belief-states-in-transformers-1 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Saturday · 52 min
“Against SURA” by Scott Alexander
The AI Futures Project distinguishes several different plans for navigating the AI transition, which I'll lump into two broader categories: Plan A (my combination of their Plan A/S/Other) involves working for a global pause on AI development, hopefully lasting decades. During this pause, we either research preparatory technologies like alignment or human intelligence amplification, or take actions to "fortify" the pause so that it can last indefinitely. Plan C (my combination of their Plan B/C+/C/sort of D) is for a world where we fail at getting a global pause, and race dynamics continue to dominate. If the leading AI companies are alignment-pilled, they coordinate a mutual pause among themselves, lasting the length of their joint lead over the leading non-alignment-pilled AI company. Then they "burn the lead" researching alignment, and release hopefully-aligned superintelligence at the last possible moment before the racing non-alignment-pilled AI companies beat them. The exact calculus changes depending on who is in the lead when - is there one alignment-pilled AI company with a six-month lead in front of everyone else? Are there two alignment-pilled AI companies with a joint two-month lead in front of their nearest competitor? Etc. The point is, if it's impossible to [...] --- Outline: (06:28) I. Questioning Assumptions Behind SURA (19:42) II. Actually, most of this stuff is bad for a global pause, too. (28:17) III. What incentives are we fighting here? (43:33) IV. What if we survive? (47:01) V. What's The Alternative To SURA? The original text contained 16 footnotes which were omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/xikGZ8fuhdG6DApAo/against-sura --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Saturday · 5 min
“What Social Constructs Are (And Are Not)” by johnswentworth
This post is mostly adapted from a section of another post. I wanted a standalone version, to serve as background for a sequence on the mechanisms of social reality. You’ve probably heard that things like laws, conventions, norms, cultural associations, status, and so forth are “social constructs”: they’re kinda arbitrary in some sense, an artifact of social reality as opposed to physical reality. This post offers a little more clarity on what social constructs are, and the ways in which they are and are not “arbitrary”. Example: Vehicle Categories The vehicles humans build today mostly fall into categories like “minivan” or “truck” (among other categories). These categories cash out in predictions: when I see a vehicle with sliding side doors, I predict that it will have a trunk door that opens overhead, three rows of seats, be driven by someone with kids, etc. On the other hand, when I see a vehicle with a flat bed in the back, I predict that it will not have three rows of seats, will have a back gate which opens downwards, be driven by my cousin, etc. But humans could choose to make different vehicles, which would change which clusters exist [...] --- Outline: (00:43) Example: Vehicle Categories (03:34) Example: Laws (04:52) The General Principle The original text contained 1 footnote which was omitted from this narration. --- First published: October 31st, 2026 Source: https://www.lesswrong.com/posts/aHqKqTcvLEkMNkAHX/what-social-constructs-are-and-are-not --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 1–20 of 537 episodes