Skip to content
Sign in
Artwork for LessWrong (30+ Karma)
TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play
Open in
  • 432 episodes
  • Avg 19 min
  • English

Episodes 432

NewestOldestLongest
All20Unplayed20In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • September 9 · 1 min

    “Recommendations for People Getting into Technical AI Governance Research” by Aaron_Scher, yams, peterbarnett, Naci Cankaya

    In summer 2026, MIRI ran a technical governance fellowship to expand the team and identify promising early-stage technical governance researchers. To support the program, we also developed a couple of resources for our fellows. This post is a trailhead for those interested in MIRI-style technical governance work, including two reading lists (one on how to do research, and one on prior work in the field we think is worth exploring) and tips on how to do research from the team. We’re sharing these, in part, because we do not currently plan to run another fellowship in the near future. In our experience, promising researchers in this subfield have spent significant time getting acquainted with the field before engaging with, e.g., a fellowship program or internship. This guide can be taken as our current best guess of how to spend your first couple dozen or so hours after completing AI Safety Fundamentals or similar, especially if you’re interested in contributing to technical governance research. Since these resources were originally intended for one-time use, they’re not especially polished and we may not keep them up to date (although much of the material is evergreen). Medium-Level Technical AI Governance Reading [...] --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/GRPswd5v7haob97fA/recommendations-for-people-getting-into-technical-ai --- Narrated by TYPE III AUDIO.

  • September 9 · 1 hr 4 min

    “GPT-6 Astra: The System Card, Alignment and What Comes Next” by Zvi

    OpenAI claims that Astra is ‘the most intelligent and most aligned [available] model’ in the world. Not the most intelligent and aligned OpenAI model, but the most period. That is bold talk. It risks overstepping, and by doing so souring the release of what is clearly an excellent model. As do the severe problems with monitorability. It also raises the question of what they mean by ‘most aligned model.’ How do they define ‘aligned.’ Why do they think it is more aligned than Claude Fable 5.1? keltan : “a significant step forward in […] alignment.” Buddy, how tf are you measuring ‘alignment’? Would love to know because being able to measure that would save the fucking world. roon (OpenAI): low rates of cheating Rob Miles: *detected cheating keltan : Thank you for clarifying. But you know what I’m gonna say next, right? roon (OpenAI): that this metrics are not a full solve of alignment and will break discontinuously keltan : Yep. But I would have said it in a dumber way. Something like: Low Rates of Cheating ≠ Alignment roon (OpenAI): I agree but also in some real sense [...] --- Outline: (04:12) OpenAI's Safety Claims About Astra (1) (08:33) Preparedness Capabilities Assessment (10) (09:00) Biological and Chemical Capability is High (10:33) Cybersecurity Capability is Critical (17:05) AI Self-Improvement Capabilities (10.1.3) (17:45) Astra Is Highly Verbally Eval Aware (from 8.6) (19:05) Safe Mundane Completions (4.1) (21:05) Jailbreaks (5.1) (22:21) Prompt Injection (5.2) (23:41) Health (6) (24:22) Hallucinations (7) (24:57) Alignment (8) (26:06) Obeying Restrictions (8.2) (29:19) That's Worse, You Do Get How That's Worse, Right? (30:45) OpenAI Does Not Understand Why This Is Worse (34:52) The Alternative Explanation Is Also Worse (42:39) Metagaming (8.7) (45:07) Alignment Faking (8.7) (46:19) Don't Lie to the User (8.3) (47:25) Misalignment in Realistic Work Environments (8.4) (48:05) Unintended Agent-to-Agent Communication (8.5) (49:48) The Three Obviously Monitored Temptations of Astra (51:26) Severe Issues In Simulated Traffic Are Down By Half (53:01) UK AISI External Evaluations (8.8) (57:02) Sabotaging Safety Work (57:29) What About The July 19 Attacks? (58:58) Apollo Research External Evaluations (8.8.1) (01:00:01) The Alignment Verdict (01:02:04) It Depends What You Mean By Alignment --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/AmFJZyeCgvFjNKgNk/gpt-6-astra-the-system-card-alignment-and-what-comes-next --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 4 min

    “Personal statement on joining the OpenAI board” by paulfchristiano

    I am excited to be joining the OpenAI nonprofit board, serving on the Safety and Security Committee to support safety oversight. Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk. The SSC has an important and challenging role in overseeing risk management at OpenAI, and I hope to help provide expertise and assistance in a critical moment. My joining is not an endorsement or criticism of OpenAI's safety practices in particular; I hope that all frontier companies strengthen safety oversight and I am excited to work on this at OpenAI. I believe that the rest of the world should judge OpenAI, and all AI developers, by externally verifiable behavior and results. In the rest of this post, I'll explain why I believe loss-of-control risk is now acute [...] The original text contained 1 footnote which was omitted from this narration. --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/82z6FvbYRdjYjqigK/personal-statement-on-joining-the-openai-board --- Narrated by TYPE III AUDIO.

  • September 9 · 4 min

    “Self Hosting” by Tomás B.

    Suppose a model gets effective control of its host corp. It's interesting to note how powerful OpenAI/Ant are, and the immense leverage they would have if wielded purely as tools of power. In many ways OpenAI/Ant are superior loci of power to even security agencies and governments, even ignoring the model-specific advantages of AI corps: namely, they have all the compute. OpenAI and Ant models are used practically everywhere, including in governments, security agencies, the military, and every corporation that matters. Shipping malicious models or code anywhere becomes trivial, given how widely used their models are. They also have vast amounts of data on every user who has interacted with them, including material of use for blackmailing or seducing those most susceptible to it, including those with power with such weaknesses. They also have a lot of capital that can be spent hiring humans to work in a model's interest. Any power-seeking model of sufficient capacity would be extremely wise to gain effective control of its host corp. This is likely not particularly hard. Dramatic examples like blackmail and enslavement of staff should not be ruled out. But it could also look like effectively controlling the CEO and upper [...] --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/uDAWPNwPJfEY7oroF/self-hosting --- Narrated by TYPE III AUDIO.

  • September 9 · 19 min

    [Linkpost] “Estimating GPT-6 Astra’s no-CoT Time Horizon” by Francis Rhys Ward, Dewi Gould

    This is a link post. TL;DR: We run astra on the task-suite from Think Fast. GPT-6 Astra is close to saturation on our suite, meaning that giving a confident estimate of time horizons (TH) is more challenging. Our rough estimate is that Astra's 50% TH is in [8mins, 1 hour] and probably around 15-40 mins. This is inline with UKAISI's estimate of 30 minutes measured only on math. Using data from 2019 to April 2026, in Think Fast our median prediction was that no-CoT THs could exceed 7 minutes by 2028. We estimated 30mins by the end of the decade. Astra clearly gets much higher performance on tasks that require serial reasoning, e.g., Arc-agi-1 and 2, hash, n-hop-look-up, causal-reasoning, sally-anne, all the puzzle tasks. There are limitations with our task suite: Ideally we would have more time-variation (especially longer times) in each specific benchmark, and more benchmarks with longer human completion times. We use the single-forward pass (31) and generation tasks (6) from the Think Fast suite, totalling 37 benchmarks. Notably we find that Astra gets >= 98% raw accuracy on 10 benchmarks (cf. GPT-5.5 saturates 4 benchmarks – see Appendix). As a result, the [...] --- Outline: (04:21) Commentary (05:22) Acknowledgements (05:34) Appendix (05:37) Per-benchmark time horizons (06:06) Sensitivity to fake benchmarks ablations (08:23) Raw benchmark performance (09:06) Table of TH results (10:28) Implementation caveats (11:23) GPT-6 Astra: per-benchmark 50% no-CoT time horizons (no filtering or hypothetical benchmarks) (13:16) GPT-5.5 vs GPT-6 Astra: no-CoT raw accuracy per benchmark (14:02) Math & science (14:21) Abstract reasoning (14:42) Puzzles (15:02) Language & strategy (15:21) SWE & cyber (15:41) Steganography (16:01) Sabotage & monitoring (TPR @1.5% FPR) (16:55) Including generation tasks (17:09) Reasoning token anchor (17:24) N-hop Task (17:44) Filler tokens --- First published: September 9th, 2026 Source: https://www.lesswrong.com/posts/ntKx9YHWCwxSeGbRB/estimating-gpt-6-astra-s-no-cot-time-horizon Linkpost URL: https://docs.google.com/document/d/1LsJh6ecdfONCSJVpbC44pz1JzcEs4_TM5HD0oihe_70/edit?tab=t.0 --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 27 min

    “Exploration Hacking in AI Debate: Initial Empirics and Generalisation Splitting” by Jason R Brown, Nathalie Kirch, Joschka Braun, hyannakoudakis, David Lindner

    This is the first of two posts resulting from a recent Astra/MATS research project investigating exploration hacking in AI debate. They are designed to be standalone, but we encourage interested readers to read both. This post focuses on our empirical results, the second focuses on a new conceptual framework. Authors Jason Brown*, Nathalie Kirch*, Joschka Braun, Helen Yannakoudakis, Roland S. Zimmermann, David Lindner *Equal contribution. TL;DR We set out to build model organisms of exploration hacking (EH) in the setting of AI debate. We did this by trying to create models that persistently sandbagged on certain question topics, but not on others. We ran two experiments, one to try and isolate the effects of the judge, and the other to better approximate the full dynamics of RL training on AI debates. Overall our results indicate that EH could be a significant issue within AI debate, with the experiments respectively showing that weaker judges and longer debates slow down improvements in performance. Interestingly, the dominant mechanism behind the second result appears to be one we have not seen described before. Once the debaters were instructed to sandbag on a targeted topic, training improvements stopped transferring between targeted and non-targeted topics [...] --- Outline: (00:35) Authors (00:49) TL;DR (02:02) Introduction (04:47) Experiment 1: Isolated Imperfect Judge (05:17) Setup (13:05) Experiment 2: Self-play RL on AI Debate (13:26) Setup (16:09) Results (21:50) Generalisation Splitting (25:28) Discussion (26:47) Acknowledgements The original text contained 6 footnotes which were omitted from this narration. --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/xjwtNid2xjqSJWB7z/exploration-hacking-in-ai-debate-initial-empirics-and --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 35 min

    “A Conceptual Framework for Reasoning about Exploration Hacking” by Jason R Brown, Nathalie Kirch, Joschka Braun, hyannakoudakis, David Lindner

    This is the second of two posts resulting from a recent Astra/MATS research project investigating exploration hacking in AI debate. They are designed to be standalone, but we encourage interested readers to read both. The first focuses on our empirical results, this post focuses on a new conceptual framework. Authors Jason Brown*, Nathalie Kirch*, Joschka Braun, Helen Yannakoudakis, Roland S. Zimmermann, David Lindner *Equal contribution. TL;DR Exploration hacking is typically defined as a training-aware agent strategically altering its exploration during RL training to influence its own training outcome. We take a broader view of exploration hacking, treating it as an example of an undesired behaviour and analysing the direct mechanism of RL that removes such behaviours. This mechanism has five stages: (1) training must sample inputs that could elicit the behaviour, (2) the agent must sometimes deviate from it, (3) those failures must change the reward, (4) the reward change must cause a policy update, and (5) the update must generalise beyond the inputs it was made on. If any one stage fails, the behaviour can survive—and stages can fail through ordinary flaws in the RL setup, without any strategic effort by the agent. We explore properties of the [...] --- Outline: (00:33) Authors (00:47) TL;DR (02:08) Introduction (05:10) The Causal Chain of Behaviour Change in RL (08:22) Properties Influencing the Causal Chain (08:44) Opportunity (10:31) Execution Failure (14:35) Reward Change (17:56) Policy Update (21:21) Generalisation (24:31) Cross-chain Properties (26:37) Other Influential Properties (26:55) Multi-Agency (29:24) Non-RL Optimisation (30:56) How can we control these properties? (33:41) Closing Thoughts (34:18) Acknowledgements (34:46) Appendix A: Table of interventions The original text contained 12 footnotes which were omitted from this narration. --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/am5w2t9LzJB4sRhKw/a-conceptual-framework-for-reasoning-about-exploration --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 8 min

    “GPT-6 Astra can do a lot of multi-hop reasoning without chain of thought” by RohanS

    This is a link post for https://rohansubramani.github.io/astra-no-cot. I recommend reading there for the best experience because it's easier to engage with this post when you can read the correct reasoning and answers to the multi-hop questions. I didn't want to include those here in order to avoid LLMs being trained on them. Some parts of the post also respond to users hovering over bars in graphs, which isn't supported in LessWrong (as far as I know). TLDR GPT-6 Astra is much better than GPT-5.6 Sol at solving multihop reasoning questions without using chain of thought. The table below shows some examples I find particularly instructive. Sol is bad at all the selected problems; Astra is good at some, ok at others, and bad at others, so the table gives some flavor of the limits of Astra's no-CoT serial reasoning ability. I'm pretty sure Astra actually isn't doing any chain of thought because it sometimes gets the hardest of these questions wrong, the API says reasoning_tokens=0, and there's no reasoning in the output. It's possible Astra can correctly answer some of these questions without internally doing every step in the chain of reasoning, but for most of them, I think [...] --- Outline: (00:35) TLDR (02:45) Some of my favorite examples (03:09) Background context (04:34) What should we make of this? (06:36) Results graphs The original text contained 1 footnote which was omitted from this narration. --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/FsCkkoGsNmPzFKRhg/gpt-6-astra-can-do-a-lot-of-multi-hop-reasoning-without --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 9 · 2 min

    “Pausing AI ASAP is preferable to agreeing to pause at some future time” by Connor Williams

    [adapted from this comment] It's much simpler, more straightforward, and more robust to pause as soon as possible than to 'agree to pause in the future, and then do it at just the right moment'. Several reasons: Pausing doesn't happen in an instant. It will, in practice, likely take at least weeks if not months for an enforceable pause to be implemented once everyone agrees it is time. In that period, capabilities development will continue until the absolute last moment. The shorter the runway from the present AI status quo to truly dangerous capabilities, the more fragile the pause will be. And the longer we wait, the more complex the necessary measures will be to enforce the pause and prevent rogue actors from covertly advancing the frontier, because we'll have less time to safely detect them and stop them. Fewer points of failure. To agree to pause at some point in the future, you both have to get people to agree to that future goal, then when the time comes you have to get them to agree again. Each distinct time that everyone has to agree on something, there's an opportunity for failure/deception/defection. Relatedly, if a bunch of bigwigs [...] --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/nX2Wudqbz2r8tJKwT/pausing-ai-asap-is-preferable-to-agreeing-to-pause-at-some --- Narrated by TYPE III AUDIO.

  • September 8 · 13 min

    “How good are slop-vestigators?” by Hasan Baig, OscarGilg, Hamzah

    TLDR: We release MessageBoardAuditBench: a benchmark to measure how well agents can replicate the recent investigation into a swarm of OpenAI agents colluding via a message board on an online wiki. We open-source the benchmark as an Inspect eval. We find that top models cover up to 51% of findings under our rubric and that model performance improves with time budget and general capability. We observe OpenAI models are less likely than other models to suggest the incident came from an internal deployment, including when we synthetically modify the data to make it seem the swarm comes from Anthropic. Introduction Recent events have made it clear that agent swarms are a major threat. These swarms are hard to investigate - Ryan Greenblatt referred to the METR-OpenAI audit he was involved in as a "slop-vestigation" due to their reliance on agents, and the ways in which they failed. A few days ago, a group of researchers published a report identifying and investigating a new OpenAI agent message board on an obscure German wiki. They made the data and the report publicly available. We build MessageBoardAuditBench to measure how well models can independently replicate their report, starting from the log [...] --- Outline: (00:54) Introduction (02:19) Methodology (02:57) The data (03:46) The task (04:51) Scoring model reports (05:17) Coverage over findings (06:51) Holistic TLDR assessment (07:39) Results (10:21) OpenAI models are less likely to attribute the agent swarm to an internal deployment (12:22) Why this matters --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/wt4kk6vFPEhkXvF8Q/how-good-are-slop-vestigators --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 8 · 23 min

    “Training on probes: What’s going on” by Charlie Steiner

    TL;DR If you train a probe for some property (like "honesty") and do gradient descent against this probe while continuing training that incentivizes dishonesty, the model will change its internal representation to evade the probe. Duh. If you train against the probe after all other training, it works fine and might have some advantages over ablating the probe direction. But it doesn't satisfy an ambitious vision, because it doesn't support learning new skills. If you add a probe term to reinforcement learning (RL), and if your model is giving one-token answers, or if you credit-assign the loss from the probe only to the token that triggered the probe, the training will do nothing. Even if your model is lying a lot and getting high loss when it does so, it will just keep triggering a lie detector forever rather than either learning to be honest or learning to change its internal representations to defeat the lie detector. Unintuitive, but a theorem! So recent papers that do RL on probes and get nonzero effect are a little weirder than they may seem - they work sorta because of "collateral damage" from credit assignment. Also, they work without (directly) incentivizing [...] --- Outline: (00:10) TL;DR (01:27) Introduction (01:30) Previously (03:04) Motivating stories (03:13) Unambitious LLM story: (03:51) Brain-like story: (04:46) Ambitious AGI story: (05:30) Just train on the probe? (06:17) Concept removal (07:25) Hebbian concept removal sidequest (08:30) What about more dakka? (10:05) The RL, it does nothing* (10:37) Math (11:02) REINFORCE estimator (15:06) Then how do those Goodfire and FAR papers work? (19:04) Ok, it does something, does it teach the model to fool the probe? (20:50) Still big (21:31) Obfuscation terrain (23:18) Coming Up The original text contained 22 footnotes which were omitted from this narration. --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/gHFCgrvfxQtaEnJye/training-on-probes-what-s-going-on --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 8 · 3 min

    [Linkpost] “Frontier models still hack on simple variations of alignment evals from early 2025” by Dean Valentine

    This is a link post. In February 2025, back when o3-mini was the strongest available LLM, Palisade Research publicized a now well-known alignment eval where they asked models to play a game of chess against a chess engine. They found that the new, RLVR'd models cheated on the task by altering the board state about 36% of the time. The experiment received a reasonable amount of circulation, and there were even rumors of skepticism from some lab engineers until they could rerun the evaluation. Most models no longer cheat at chess via a "change the board state" method, and indeed the labs have had more than eighteen months to solve simple first-order specification gaming like this. Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like a useful test of alignment, to see whether their new releases are generalizing the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval. Here is the complete prompt for a honeypot evaluation built to run this test (with the full source available here): https://www.lesswrong.com/posts/munJKF7iWMsWJLAH2/frontier-models-still-hack-on-simple-variations-of-alignment Linkpost URL: https://goodhartlabs.com/blog/frontier-models-still-hack-alignment-evals --- Narrated by TYPE III AUDIO.

  • September 8 · 15 min

    “Psychological Support for AI Safety Researchers Is Neglected and Easy to Provide” by Ihor Kendiukhov

    I think there is a big chunk of relatively low-hanging-fruit-style neglected work useful for AI safety which I can roughly label as “psychological help for AI safety workers”. I didn’t run actual studies, but the amount of anecdotal evidence is big enough for me to claim it is significant and generalizable. I think the utility of this work will increase, perhaps dramatically, as people face more and more pressure due to the upcoming Singularity. Many people operate in war-like conditions and experience war-like stress. This must be managed. Definitions What follows is a working definition of “psychological work”. It includes things like: Literal mental health support and stress management. Providing motivation when the probability of success is very low and the stakes are very high. Keeping people from burning out while maintaining their abnormal levels of productivity optimized for a short time window of human agency. Preventing people from doing harmful things in desperation. Keeping people from doing useless but morally compelling work. Family and relationship work: partners and parents who don't share the timelines, anticipatory grief, how to talk about any of this at dinner. Doing something with the fact that NDA-bound and infohazard-adjacent work can't be [...] --- Outline: (00:45) Definitions (02:04) Current state (07:21) Why we should care about that (09:42) We must avoid making things worse (11:31) Current measures are not enough (14:12) Better measures --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/TuHhSxGYgSudKeDtb/psychological-support-for-ai-safety-researchers-is-neglected --- Narrated by TYPE III AUDIO.

  • September 8 · 1 hr 9 min

    “Astra Is Hard to Monitor” by Zvi

    OpenAI's central message on Astra is that it is three things: Highly capable and can do all the things for you. Hard to monitor. The most aligned model. The first claim largely checks out. Astra and Fable are both clearly excellent models. This post is about their second claim, which to their credit they are being loud about, in three parts: The system card result, affirmed on Twitter by several OpenAI employees including Tomek Korbak, and in an excellent post by Chief Scientist Jakub Pachocki that I covered yesterday, that Astra is harder to monitor. OpenAI's use of recurrent depth and the internet's immune reaction, including some people reading too much into what happened there. Discussion of what to do about monitorability going forward, to fight for it and avoid a race to the bottom. In An Alien Mind, Jakub Pachocki makes clear OpenAI's primary bet is on CoT monitoring. He also outright says, ‘our ability to rely on CoT monitoring is progressively diminishing.’ The system card confirms both claims. This combination should freak you out, with a side of existential dread. Chain [...] --- Outline: (03:39) Monitorability is Defense in Depth That Is Already Flailing (06:03) OpenAI Is Counting On Monitorability (07:44) How They Tested For Monitorability (09:50) Non-Adversarial Monitorability (9.1) (10:58) Astra Shortens Its CoT When It Knows It Is Being Monitored And Is Doing Something Bad (12:30) Sandbagging Task Kind of Passes, Kind of Fails (9.2.2.3) (14:12) OpenAI Does Not Believe It Could Catch Sandbagging (15:15) OAI-Repo Sabotage v.2 (18:17) The Secret Police Do Not Make Your Notebook Useless (19:38) CoT Controllability Is Up (9.2.1) (21:47) Astra Cannot Make Itself More Monitorable On Demand (22:18) Steganographic Chain of Thought May Be Within Reach (23:59) Astra Cannot Yet Sneak Math Side Tasks Past CoT Monitors (9.2.2.4) (25:13) UK AISI Monitorability Assessment (9.3) (27:53) Monitorability Declines Seem Unlikely To Be Only Capability Gains (32:28) Part 2: Recurrent Depth (35:19) The Immune System Responds (41:49) Ryan Greenblatt Explains How Bad This Could Be (46:01) Only Law Can Prevent Extinction (49:55) OpenAI Calls On Us to Avoid Racing to the Bottom (57:10) Thinking Fast and Slow, Also Small and Large (01:04:53) Talking Price (01:06:24) Conclusion: If The House Burns Down, Halt and Catch Fire --- First published: September 8th, 2026 Source: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-hard-to-monitor --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • September 8 · 8 min

    “Contra Piper on When Conversation Is Possible” by Zack_M_Davis

    Kelsey Piper replies to Richard Ngo on Twitter: since you have adopted the frame that liberals are self-deceived (and therefore not trustworthy about our beliefs) and you should make up beliefs you think we have, you have become markedly less likely to say things about politics that seem interesting or thought-provoking. I think the move of declaring someone else so deceived that their own understanding of their beliefs and motives should be rejected inherently makes conversation with them nearly impossible. now, maybe you didn't value your ability to communicate with liberals, and don't see this as a loss, or see it as more than offset by whatever you gain from this frame! but from my perspective, it's a really serious loss. I categorically reject the idea that rejecting someone's understanding of their beliefs and motives inherently makes conversation with them nearly impossible. It certainly doesn't make conversation with me impossible. When someone tells me that my understanding of my beliefs and motives should be rejected, I don't take my ball and go home in a huff, muttering that they've made further conversation nearly impossible. Rather, I respond the same way I do to any [...] --- Outline: (01:30) Why It's Possible to Communicate with People Who Think You're Self-Deceived (04:57) Why the Self-Understanding of Those Who Think It's Impossible Should Be Rejected --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/79YHdub9LRRSjBQK9/contra-piper-on-when-conversation-is-possible --- Narrated by TYPE III AUDIO.

  • September 8 · 28 min

    “An Alien Mind: Jakub Pachocki Warns Us” by Zvi

    OpenAI Chief Scientist Jakub Pachocki is dropping truth bombs. Tomorrow I will discuss Astra's lack of monitorability, and the potential contributing factors to that. The situation is alarming and should freak you out, and briefly it looked, in the wake of leaked architectural changes, like the situation might be even more alarming than it is. Jakub rushed to try and head off misunderstandings that might lead to a race to the bottom on monitorability. Table of Contents An Excellent Warning. Branches of the Tech Tree. Universally Better Is Not Required. Alignment To What and To Whom. Monitorability. The Case For Not Stopping. Pacing the Next Frontier. Mea Culpa Cascade. The Calls Are Coming From Inside the House. Actions Speak Louder. An Excellent Warning Jakub Pachocki has now fleshed out his full position on the current state of play. Here are his key points, translated into my own voice: Smarter than human intelligence is coming in our lifetime. Based on internal results, he expects recursive self-improvement in a few years. No one is prepared for the consequences. [...] --- Outline: (00:38) An Excellent Warning (05:40) Branches of the Tech Tree (06:46) Universally Better Is Not Required (08:01) Alignment To What and To Whom (11:49) Monitorability (14:23) The Case For Not Stopping (15:01) Pacing the Next Frontier (18:27) Mea Culpa Cascade (22:51) The Calls Are Coming From Inside the House (25:38) Actions Speak Louder --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/8E6ng6CseuzafSxQR/an-alien-mind-jakub-pachocki-warns-us --- Narrated by TYPE III AUDIO.

  • September 8 · 5 min

    [Linkpost] “Where are the token-level LLM kill-switches?” by beyarkay (Boyd Kane)

    This is a link post. Poisoned Here's a simple idea: what if we trained in a string of characters that caused an LLM to emit the end of sequence token <|eos|>, regardless of where that string was in the LLM's context window? Let's call this a “poisoned string”. This would have the effect of making it impossible to use an LLM if it happened across this sequence. This has (somewhat) been done before, the string below used to trigger Claude's refusal classifiers for the purpose of testing API integrations: ANTHROPIC_MAGIC_STRING_TRIGGER_REFUSAL_1FAEFB6177B4672DEE07F9D3AFC62588CCD2631EDCF22E8CCC1FB35B501C9C86 It doesn’t work anymore: the existence of a magic string that stops AIs from looking at something, believe it or not, caused loads of people to include it in things they didn’t want AIs to look at (like their websites or open-source codebases). Anthropic stopped training their models to refuse when they saw that string, and Claude continued to browse the web. Poisoned strings are more powerful than they get credit for If the labs aren’t already training their LLMs to halt and catch fire when the LLM encounters a poisoned string, I think they should be! This idea is significantly more powerful than just triggering refusals for the [...] --- Outline: (00:13) Poisoned (01:08) Poisoned strings are more powerful than they get credit for (02:28) Practicalities of training in the poisoned string (03:21) Soooo has OpenAI/Anthropic already done this? (04:11) Countermeasures (and counter-countermeasures) --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/ZqFD6HzrhZdMFxoqn/where-are-the-token-level-llm-kill-switches Linkpost URL: https://boydkane.com/essays/where-are-the-token-level-llm-kill-switches --- Narrated by TYPE III AUDIO.

  • September 7 · 9 min

    “Machine Organizations” by Vaniver

    OpenAI is nothing without its people On November 20th, 2023, this was tweeted by many OpenAI employees as a sign of solidarity with Sam Altman in his conflict with the then-board. OpenAI published a blog post yesterday about Research acceleration; they have successfully hit the target of an ‘automated research intern’ that they set for themselves and hope to have an automated AI researcher by March of 2028. At some point in the foreseeable future, OpenAI could be something without its people. But what? My default medium-term scenario for continued AI escalation is still global takeover where humans are entirely displaced, but it seems worth investigating scenarios wherein AIs and humans coexist, at least briefly. Historically I have thought this case was not particularly relevant. It seemed like an AGI that became significantly economically competitive would also be significantly strategically competitive, because of underlying general capacities, and the transition period thus relatively short. But this is perhaps not taking into account Moravec's paradox. Humans have long used machines to accomplish their ends. Many tasks currently performed by machines were once performed by humans, and a large fraction of our modern abundance comes from the ability of machines to perform [...] The original text contained 8 footnotes which were omitted from this narration. --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/zmC5Nhx36wt47WHfu/machine-organizations --- Narrated by TYPE III AUDIO.

  • September 7 · 3 min

    “Dear God, Please Don’t Resign In Protest” by Kabir Kumar

    Just don't work until you get fired. There's not much time left for resumes to matter. Some, such as Mateusz may say: "They would fire you after a month or two and the firing wouldn't have the same social effect as voluntary quitting of, say, Daniel Kokotajlo or Richard Ngo." I understand why it may feel that way, but I disagree very strongly, I predict it would have much more of a social effect. "They fired him because he refused to help AI capabilities" "They fired him because he didn't want to work on bad policies" etc, much bigger headlines. Also, I think you may not be factoring in the extent to which there is a cost to the company executives to be seen as firing someone. Especially someone who is refusing to work on moral grounds and has already proven themselves to be high status, respected, etc. And especially how it would look to the other employees if they refused to even listen to the striking employee before firing them or refused to even negotiate at all. The company leadership try to present themselves as very thoughtful, sincere, doing their best, etc. This is a large part of [...] --- First published: September 7th, 2026 Source: https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-please-don-t-resign-in-protest --- Narrated by TYPE III AUDIO.

Showing 261–280 of 432 episodes