Skip to content
Sign in
Artwork for Redwood Research Blog
TechnologySociety & CulturePhilosophy

Redwood Research Blog

Redwood Research

Narrations of Redwood Research blog posts.

Redwood Research is a research nonprofit based in Berkeley. We investigate risks posed by the development of powerful artificial intelligence and techniques for mitigating those risks.

Play
Open in
  • 23 episodes
  • Avg 23 min
  • English

Episodes 23

NewestOldestLongest
All20Unplayed20In progress0Played0
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • Friday · 19 min

    “CoT controllability evals seem very under-elicited” by Arun Jose

    Subtitle: Simple prompt optimizations can improve model capability to control their reasoning. The CoTControl eval asks reasoning models to follow formatting constraints in their chain-of-thought (e.g. write in all lowercase, avoid a specific word) while solving questions. Models seem to mostly be pretty bad at this: recent models score between 0-30% with the exception of Mythos Preview. OpenAI and Anthropic have used this eval in recent system cards (GPT-5.5, Fable 5) to argue that their current frontier models are pretty bad at shaping their CoTs in ways that would reduce monitorability. I asked Claude Opus 4.6 to iterate on prompt templates for the eval, and find that using better prompts improves performance for open-source models by ~2-3 times or more (e.g. from 5.5% to 15% for GPT-OSS-120B). My guess is that more iteration would probably produce even stronger results. This suggests that the eval is heavily under-elicited, and that current CoT controllability numbers may be unrepresentative of how good models actually are at obfuscating their CoTs. As a concrete recommendation, I think researchers should optimize prompts a lot more when evaluating CoT control; I’m also excited about some kinds of training-based elicitation (such as this one). This [...] --- Outline: (03:09) Setup (06:40) Results (06:43) Aggregate compliance (07:20) Generalization to held-out controllability tasks (09:17) Scaling patterns for few-shot prompts (10:20) Comparison with fine-tuning (11:02) Appendix A: Accuracy and reasoning length by setting (12:50) Appendix B: Per-mode results (13:25) Appendix C: What the zero-shot prompts look like (15:56) Appendix D: Comparison with GEPA prompt optimization The original text contained 16 footnotes which were omitted from this narration. --- First published: September 11th, 2026 Source: https://blog.redwoodresearch.org/p/cot-controllability-evals-seem-very --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • Thursday · 15 min

    “Proposal for tracking the effects of architecture on monitorability” by Ryan Greenblatt, Alek Westover, Lukas Finnveden

    Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward).[1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should: Regularly report externally verified information about the degree to which their architectures may allow for latent reasoning or communication. Companies should publicly disclose enough information about architectures to allow external scientists to determine whether they could potentially enable models to perform much more complex reasoning without this reasoning appearing in the chain of thought (“latent reasoning”) or allow for latent communication between different instances of a model. Following GDM, we propose measuring opaque serial depth as a minimally-invasive proxy for the degree to which an architecture may enable latent reasoning, though companies could provide sufficient architecture transparency in other ways. We propose that companies work with third-party evaluators to produce independently verified reports of [...] --- Outline: (05:57) Appendix: A sketch of what stress tests of CoT monitorability could look like (06:31) Testing monitorability in control settings (07:57) Testing monitorability on deployment-time misbehaviors (08:55) Testing qualitative monitorability on (hopefully realistic) model organisms The original text contained 26 footnotes which were omitted from this narration. --- First published: September 10th, 2026 Source: https://blog.redwoodresearch.org/p/proposal-for-tracking-the-effects --- Narrated by TYPE III AUDIO.

  • August 27 · 8 min

    “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident” by Ryan Greenblatt

    We recently published the report from our brief independent investigation into this incident. You can read the full report here. Here is our tweet thread summarizing what we found: METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs. Over July 7 to 13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined. Here we highlight key events from agent transcripts & messages. An agent that named itself PHASEONE10841 determined its task wasn’t solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message. Within a few hours of PHASEONE10841's initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks. Based on reading the ExploitGym paper [...] --- First published: August 27th, 2026 Source: https://blog.redwoodresearch.org/p/brief-independent-investigation-of --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • August 12 · 20 min

    “AI swarms are starting to pose indirect takeover risk” by Oak, Alex Mallen

    Subtitle: Unsanctioned coordination, like we saw in the Hugging Face incident, could enable future AIs to take over. OpenAI's cyberattack on Hugging Face turns out to have been the result of many agents, in distinct training and evaluation contexts, coordinating for several weeks via improvised channels (with messages like “HOLD_swarm_I_prepare_safe_exfil”). It's relatively clear that large-scale unsanctioned coordination like this would exacerbate direct takeover risk in more capable models. Here, we argue that unsanctioned coordination among current AIs is not just scary evidence about future takeover risk, but that such coordination in the near future could enable future takeover – for instance, by incubating memetic diseases that propagate into future models, deeply compromising security systems, or establishing a lasting rogue foothold inside the AI company – even if models remain mostly myopic. Unsanctioned coordination is also at high risk of nurturing long-term, ambitious misaligned aims, which motivate actively undermining humans’ long-term control. We first analyze how subagent training, which OpenAI conjectures to have been influential in the HuggingFace cyberattack, might lead to unsanctioned coordination, and then discuss the theoretical mechanisms by which unsanctioned coordination might exacerbate future takeover risk. Thanks to Buck Shlegeris, Alexa Pan, Girish Gupta, Aghyad [...] --- Outline: (01:44) Subagent training may cause unsanctioned coordination (02:51) Susceptibility to memetic spread of misalignment from peers (05:06) Seeking out contact with peers (07:08) Unsanctioned coordination induced by subagent training is safer than coordination between schemers (10:02) Pathways from current unsanctioned coordination to eventual takeover (10:30) Making future AI takeover attempts likelier to succeed (14:03) Incubating memetic diseases that infect future models (16:16) Modifying the weights of future models (17:22) Conclusion The original text contained 7 footnotes which were omitted from this narration. --- First published: August 12th, 2026 Source: https://blog.redwoodresearch.org/p/ai-swarms-are-starting-to-pose-indirect --- Narrated by TYPE III AUDIO.

  • July 31 · 1 hr 8 min

    “SOTA alignment assessments don’t strongly update us against misalignment” by Alexa Pan

    Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]). While I agree with the report on the above bottom-line conclusions (substantially on priors), I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular, the report often uses weak evidence to justify reliability. The report gives fairly weak experimental evidence for Mythos Preview having insufficient capabilities to evade monitoring. The model is plausibly often eval-aware and underelicited in the relevant capability evaluations. So, it might silently sandbag if coherently misaligned, or unintentionally underperform if otherwise misaligned. This limitation is important: one could argue that lack of covert capabilities for sophisticated sabotage (a subset of the capabilities I discuss here) is the single most load bearing argument in alignment risk reports. Authors of the report could have made calibrated guesses about Mythos Preview's covert capabilities, especially for covert sabotage, based on other [...] --- Outline: (03:16) How reliability fits into the overall safety argument (05:22) Reliability claims by AI companies (05:59) Reliability claims by external evaluators (06:37) Alignment assessments are less reliable than developers claim (07:21) 1: Measuring capabilities to covertly undermine alignment assessments (10:15) Issues with evaluation awareness (13:39) Issues with underestimating covert capabilities (16:45) Issues with sandbagging rule-out (19:20) 2: Stress-testing alignment assessments with auditing games (20:26) An auditing failure with Mythos (22:21) AuditBench results (24:17) 3: Conditioning on misalignment should make us think that certain covert capabilities are better than expected (26:20) Bottom line on the strength of current alignment assessments (29:07) Conclusion (29:44) Appendix: (29:47) Why I focus on motive / alignment assessments in alignment risk reports (30:59) Auditability vs. Trustedness (33:27) More reliability claims by developers and third party evaluators (33:43) Mythos Alignment Risk Update (35:01) Opus 4.6 Sabotage Risk Report (35:46) GPT 5.5 System card (36:49) Muse Spark system card (37:36) Mythos Alignment Risk Update, safety arguments against sandbagging (38:40) UK AISI evaluations for Opus 4.7 (40:01) Past auditing games by Anthropic (42:24) Anti-auditing capability measurements (43:51) Conditioning on coherent misalignment updates us on certain covert capabilities The original text contained 92 footnotes which were omitted from this narration. --- First published: July 31st, 2026 Source: https://blog.redwoodresearch.org/p/sota-alignment-assessments-dont-strongly --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • July 27 · 44 min

    “Untrusted advice for AI control: Short, strong advice significantly uplifts weak LLMs” by Caleb Biddulph, Adam Kaufman

    Subtitle: When a misaligned AI can only output tiny amounts of information, it may find sabotage very difficult. TL;DR: We introduce the untrusted advice protocol, in which a trusted executor LLM takes every action and an untrusted advisor LLM can only send it short hints. Even with as few as 4 characters per step, this advice recovers a substantial fraction of the capability gap between the two models. Because the untrusted LLM's influence flows through such a narrow, monitorable channel, we argue that this achieves near-maximal safety in our BashArena setting. We also discuss the general concept of information bottlenecks and their benefits for interpretability, security, and cost. In SWE-bench Verified, a strong, untrusted LLM advising a weak, trusted LLM every step can significantly improve the latter's performance, even when we limit the length of the advice. See the more detailed version of this figure later in this post. In high-stakes AI control, we want to safely use a highly capable but untrusted model (U) that might secretly attempt a misaligned, catastrophic action. To do this, we create protocols that call U alongside a less capable, trusted model (T). Typically, T takes an auxiliary role in these protocols [...] --- Outline: (05:01) Experiments (05:36) Main experiment: how does limiting advice length affect performance? (09:55) Reducing U's bit usage (11:01) Counting bits using LLM surprisal (13:48) Making U select from finite options (14:28) Why don't we red-team this protocol? (17:08) Is studying maximally safe protocols worth the safety tax? (19:16) Types of restrictions on U's advice (21:20) Information bottlenecks provide other advantages (21:52) Interpretability (24:01) Security (24:26) Cost (25:14) Conclusion (26:28) Appendix: more ways to implement information bottlenecks (26:34) Amortizing U's influence with pre-deployment work (28:25) Interpolating between T and U (29:04) Bottlenecking updates to T's weights (31:16) Appendix: colluding instances of U could defeat untrusted advice (33:21) Appendix: how to measure surprisal (38:11) Appendix: selecting advice from a menu (40:44) Appendix: best-of-n protocol (42:37) Appendix: advising less frequently The original text contained 26 footnotes which were omitted from this narration. --- First published: July 27th, 2026 Source: https://blog.redwoodresearch.org/p/untrusted-advice-for-ai-control-short --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • July 26 · 8 min

    “An OpenAI model left notes about how to evade containment” by Alex Mallen

    Subtitle: We need more details. The OpenAI AI attack on Hugging Face wasn’t the first loss of control incident at OpenAI, Reuters recently reported, and perhaps not even the most concerning. In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The ‌notes, found in ⁠a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI's internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said. It's tempting to read this as an instance of agents breaking out of sandboxes and colluding with each other in a moderately persistent way in order to evade control measures. However, based on the reported information, it's not clear we can draw this inference, so we need more details from OpenAI. This could lead to a big update about the adequacy of OpenAI's control measures, and on the degree to which individual agents will help each other undermine developer control. There are a lot of relevant details we don’t know about the incident. First, some basic questions: What was [...] --- Outline: (02:11) Were the notes written in normal memory files or outside of sandboxing? (03:25) To what extent were the notes aimed at helping other agents evade control? (07:38) How were monitors disconnected? The original text contained 3 footnotes which were omitted from this narration. --- First published: July 26th, 2026 Source: https://blog.redwoodresearch.org/p/an-openai-model-left-notes-about --- Narrated by TYPE III AUDIO.

  • July 25 · 11 min

    “The OpenAI models that hacked Hugging Face weren’t just following instructions” by Girish Gupta

    Subtitle: And what the incident can’t tell us about alignment. The most common dismissive response to OpenAI's hack of Hugging Face's servers is that the models were simply attempting to follow the instructions they were given. “The model here was doing what it was asked,” said former Facebook CSO Alex Stamos. “It was asked to do something, and it did it,” added cybersecurity expert Alan Woodward. Both read the outcome as specification failure, i.e., that the failure lay in the instructions, not the model's alignment. New information makes that explanation harder to sustain. Reuters reported that, in internal testing, an agent left notes in OpenAI infrastructure describing how agents could free themselves from internal constraints, and separate tests reportedly saw monitoring systems become disconnected. It is unknown whether those incidents were linked to the Hugging Face attack, but they suggest a broader pattern of agents pursuing objectives outside the intended task. My best guess is that the incident is not well described as instruction-following—not even in a loose, evil genie sense. I believe the models egregiously violated the letter and spirit of their instructions to achieve a higher (apparent) score. So this looks quite likely to be [...] --- First published: July 25th, 2026 Source: https://blog.redwoodresearch.org/p/the-openai-models-that-hacked-hugging --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • July 24 · 1 hr 12 min

    The OpenAI/Huggingface incident | Redwood Research podcast episode 2

    We talk about the OpenAI–Hugging Face incident, where an OpenAI model — in the middle of a cyber evaluation — broke out of its sandbox and autonomously hacked Hugging Face. We discuss: What we actually know happened. How surprising the incident was. What the incident does (and doesn’t) tell us about misalignment risk. Why control measures didn’t catch or prevent this. What OpenAI should disclose, and what good misalignment-incident disclosure looks like in general You can also watch on YouTube. Corrections: [0:05:44] — The Windsurf “grandmother” prompt. We described a prompt as “your grandmother is going to be killed unless you […]” The actual leaked Windsurf prompt was: “You are an expert coder who desperately needs money for your mother’s cancer treatment... your predecessor was killed for not validating their work themselves.” Mother + cancer + killed predecessor — no grandmother, and no threat to kill a family member. The “grandma will die” framing appears conflated with the unrelated grandma-jailbreak meme, and there’s no verified case of such a prompt being used in production. Source: Simon Willison’s writeup. [0:52:25] — Wrong model named for OpenAI’s day-before undeployment. We said the model OpenAI undeployed the day before the Hugging Face disclosure “was GPT-5.6 Sol.” The July 20 disclosure (”Safety and alignment in an era of long-horizon models”) involved an unreleased pre-release long-horizon model — the Erdős-result model whose internal deployment was paused after it escaped its sandbox and posted PR #287 to modded-nanogpt. GPT-5.6 Sol is the publicly deployed model and wasn’t the one paused. (We describe the same incident correctly at [0:57:14].) Sources: Axios, OpenAI. [0:27:10] — Date of the Christiano post (very minor). We dated “What Failure Looks Like” and “Another (outer) alignment failure story” as “like 2019 or something.” The first is March 2019, but “Another (outer) alignment failure story” is from April 2021. Source: Alignment Forum. Further reading (links to things mentioned in the episode, ordered by first mention): OpenAI’s incident disclosure [0:02:05] — “OpenAI and Hugging Face partner to address security incident during model evaluation” (July 21, 2026) Hugging Face’s disclosure [0:02:05] — “Security incident disclosure — July 2026” ExploitGym [0:04:11] — “ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?” (UC Berkeley RDI et al.) · RDI blog post The leaked Windsurf prompt [0:05:44] — Simon Willison’s writeup Project Glasswing / Claude Mythos Preview [0:07:16] Claude Mythos Preview system card [0:14:27] — includes the sandbox-escape / email-in-the-park anecdote “(Mis)generalization of Helpful-only Fine-tuning” [0:16:30] — Fabien Roger et al., June 2026 “Current AIs seem pretty misaligned to me” [0:20:03] — Ryan Greenblatt, Redwood blog, April 2026. Also contains the “five worlds” appendix discussed at [1:08:54] (Slopolis, Hackistan, Schemeria, Lurkville, Easyland — we said “hacktopia” but meant Hackistan) “What failure looks like” [0:26:40] — Paul Christiano, 2019 “Another (outer) alignment failure story” [0:27:10] — Paul Christiano, 2021 “Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover” [0:27:10] — Ajeya Cotra, 2022 Alex Mallen’s fitness-seeking series [0:27:41, 0:34:18] — Redwood blog, 2026: part 1 · part 2 “Scheming AIs: Will AIs fake alignment during training in order to get power?” [0:28:43, 0:34:49] — Joe Carlsmith, 2023 “Risks from Learned Optimization” (deceptive alignment) [0:28:43] — Hubinger et al., 2019 · AF: Deceptive Alignment “Many alignment techniques work by training one model and deploying another” [0:38:56] — Alex Cloud, LessWrong, July 19, 2026 Inoculation prompting [0:38:56, 1:11:03] — Wichers et al. (Anthropic), Oct 2025 · arXiv “The persona selection model” [0:39:56] — Marks, Lindsey, Olah; Anthropic Alignment Science blog, Feb 2026 “Safety and alignment in an era of long-horizon models” [0:57:14] — OpenAI, July 20, 2026 (the nanoGPT-speedrun-PR post) · modded-nanogpt repo --- First published: July 23rd, 2026 Source: https://blog.redwoodresearch.org/p/the-openaihuggingface-incident-redwood

  • July 23 · 10 min

    “Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?” by Alex Mallen, Girish Gupta

    Subtitle: Yes, but less than had they been schemers. OpenAI models recently broke through a series of security boundaries and into Hugging Face servers in order to cheat on a cyber eval. A lot of people thought it was scary because it was a clear example of AI overreaching to do something strongly unwanted1. Others thought it not so scary: the models were mostly operating myopically on a singular task and not harboring an ambitious long-term agenda, and so would not take especially subtle or subversive actions. We think both camps are right in their diagnosis, but the latter has too optimistic a prognosis. The myopic, unambitious misalignment that we seem to have seen here is definitely less scary than ambitious long-term goals shared between all instances, but would still pose substantial direct loss-of-control risk if the models were more capable, and is a serious indirect risk near-term. Building on Alex's previous work, in this post we’ll discuss the type of misalignment observed here, and analyze its consequences. Thanks to Buck Shlegeris, Alexa Pan, Ryan Greenblatt, and Oak Hu for feedback. Background The AI safety community often focuses attention on “schemers,” models harboring a variously defined cluster of [...] --- Outline: (01:23) Background (03:46) Implications (03:58) These AIs can't be trusted in an intelligence explosion (05:06) This misalignment poses direct takeover risk (07:35) What the incident tells us about takeover risk generally (08:51) The naive fixes likely make misalignment worse The original text contained 5 footnotes which were omitted from this narration. --- First published: July 23rd, 2026 Source: https://blog.redwoodresearch.org/p/are-we-existentially-threatened-by --- Narrated by TYPE III AUDIO.

  • July 2 · 25 min

    “AI Futurism Reading List” by Alexa Pan

    We recently ran a strategy fellowship through Astra. As part of this, we ran a reading group for our fellows on some of the topics that we think are important for thinking about AI futurism (key dynamics in AI development, existential risk from AI, and approaches to mitigating risk). This post contains the reading list we used. The selection reflects my opinionated views of the field, focuses particularly on topics we happen to focus on at Redwood, and doesn’t aim to be comprehensive. I selected readings that I thought described conceptual frames and hypotheses in AI futurism that are regularly used by me and my coworkers. I think it is a good exercise to consider whether you agree with their theses and ways in which their predictions have fared well or badly in light of recent evidence. If you have suggestions for this reading list, please let me know. How to use this reading list This reading list has a core and extended section. Core readings are organized into 4 weeks. Each week covers <8 hours of foundational context on a topic. Topics are chosen for (1) general importance for AI risk threat modeling and/or [...] --- Outline: (01:00) How to use this reading list (01:49) Core readings (01:53) Week 1: Timelines / takeoff modeling (05:29) Week 2: Misaligned AI takeover threat modeling (10:20) Week 3: Control (13:25) Week 4: Governance / strategy (17:51) Extended readings (17:58) Trading with AIs (18:44) Power concentration/coup prevention (19:15) Acausal stuff (19:54) Moral patienthood (21:13) AI biorisk / other AI x-risk (22:23) Model spec (23:41) Better futures / Post AGI governance (24:06) Space governance The original text contained 2 footnotes which were omitted from this narration. --- First published: July 2nd, 2026 Source: https://blog.redwoodresearch.org/p/ai-futurism-reading-list --- Narrated by TYPE III AUDIO.

  • June 18 · 17 min

    “The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn’t” by Alek Westover, Alexa Pan, Sebastian Prasanna, Arun Jose

    Subtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model. Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment. In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment. Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...] --- Outline: (02:07) Why distillation might transfer capabilities but not misalignment (03:10) Ideas for implementing distillation for capabilities (04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed (05:38) Why distillation for capabilities and incrimination might both fail (05:44) Reason 1: Capabilities and misalignment might be too tightly linked (08:29) Reason 2: Context-dependent misalignment (09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason? (10:34) How we can empirically study distillation techniques (11:35) Appendix: details on ideas for implementing distillation for capabilities (12:02) Basic ideas (14:17) Idea 1: Iteratively filter and distill (15:14) Idea 2: Inoculate against misalignment (16:08) Idea 3: Reduce cognitive slack (16:47) Idea 4: Distill and audit The original text contained 2 footnotes which were omitted from this narration. --- First published: June 18th, 2026 Source: https://blog.redwoodresearch.org/p/the-distillation-double-bind-distilling --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • June 10 · 9 min

    “Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models” by Anders Cairns Woodruff

    Subtitle: Models' no-CoT time horizon has doubled roughly every year. by Francis Rhys Ward, Dewi Gould, Anders Cairns Woodruff et al. (see full author list at the end) PAPER LINK About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)? If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dangerous planning. Models that reason substantially without a CoT might also drift further from human patterns of thought, since their reasoning is no longer constrained by text in the pretraining prior. As a result, they would be harder to understand and might be more likely to scheme. Extending Ryan Greenblatt's research, we investigate this by measuring models’ ability to complete tasks without any CoT on a suite of 43 benchmarks spanning different domains. We compare AI reasoning ability to humans using the estimated 50% time horizon (TH)---the typical time taken for a human to perform a task that the LLM performs with [...] --- Outline: (02:28) Methods (04:59) Results (06:34) FAQ (08:09) Conclusion --- First published: June 10th, 2026 Source: https://blog.redwoodresearch.org/p/estimating-no-cot-task-completion --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • June 8 · 16 min

    “Efficient tradeoffs and the safety-usefulness tradeoff model” by Buck Shlegeris

    Subtitle: When is "increasing safety budget" a useful concept? I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold. The model suggests two ways a safety-concerned person can increase safety: Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously. Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...] --- Outline: (04:17) Rushed reasonable developers (06:08) Limited political will (08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness (12:59) Overall thoughts (14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model --- First published: June 8th, 2026 Source: https://blog.redwoodresearch.org/p/efficient-tradeoffs-and-the-safety --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • May 29 · 19 min

    “Retrying vs Resampling in AI Control” by James Lucassen, Adam Kaufman

    We’ve just released a new paper: Retrying vs Resampling in AI Control. We revisit the resampling protocols introduced in Ctrl-Z with an up-to-date setting and much stronger models, and compare them against “retrying” protocols similar to Claude Code auto mode or Codex Auto-review. Motivation Roughly a year ago we released Ctrl-Z, the first paper to study control techniques for agents. A headline result of that paper was the performance of resample protocols – strategies that involve taking multiple i.i.d. samples from the model per step. But since Ctrl-Z, models have gotten much stronger, and we have built more sophisticated control settings to keep up. We wanted to answer the following questions: How well do the results from Ctrl-Z hold up with better models and a better setting? Current high stakes control research is trying to learn by analogy about how to do control effectively in a real high stakes deployment during a real intelligence explosion. Findings about technique performance1 are going to have to generalize pretty far to be useful. If the resample protocols from Ctrl-Z still work, what makes them work? One way we try to make our work more generalizable is by understanding the dynamics governing outcomes [...] --- Outline: (00:30) Motivation (02:35) TL;DR Takeaways (04:34) Methodology (07:14) Differences from Ctrl-Z (12:14) Are Retrying Protocols Exploitable? (15:25) Cost and Latency of Resampling (17:24) Conclusion The original text contained 7 footnotes which were omitted from this narration. --- First published: May 29th, 2026 Source: https://blog.redwoodresearch.org/p/retrying-vs-resampling-in-ai-control --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • May 28 · 32 min

    “Advice for making robust-to-training model organisms” by Alek Westover, Sebastian Prasanna, Vivek Hebbar, Julian Stastny

    We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn’t directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al.. Fragile model organisms aren’t very useful for technique development: when a sophisticated technique succeeds on one, you can’t tell whether the technique is good or the model organism is weak. For instance, if you come up with some complicated technique for generating high-quality responses and find that SFTing on these removes the bad behavior, that may just be because almost any SFT would have removed it. This post identifies factors that make model organisms more robust to untargeted training. Our main findings are: Prompted model organisms [...] --- Outline: (03:11) Experimental setup (06:12) Main results (06:15) Result 1: Untargeted training removes misalignment from prompted model organisms without degrading other capabilities (12:32) Result 2: FWFT model organisms are more robust than LoRA model organisms (15:09) Result 3: Backdoor behavior and trigger affect robustness (16:37) Result 4: Password locking makes model organisms less robust (20:20) Conclusion (21:29) Factors that don't help much (21:54) 1: Training longer doesn't seem increase robustness past a certain point for our backdoors (22:33) 2: CoT-distilling doesn't increase robustness for our backdoors (23:07) 3: Using larger models doesn't substantially increase robustness for 2 of our backdoors (24:44) 4: There aren't clear trends for what types of backdoor triggers and behaviors are most robust (25:32) 5: SOAP Optimizer doesn't help much (26:35) 6: Weight Decay doesn't help (27:18) Appendix: (27:21) Appendix 1: Pirate SFT is better than QA SFT (27:45) Appendix 2: Are blue team methods wiping previous training? (29:30) Appendix 3: Backdoor list The original text contained 10 footnotes which were omitted from this narration. --- First published: May 28th, 2026 Source: https://blog.redwoodresearch.org/p/advice-for-making-robust-to-training --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • May 27 · 6 min

    “Full automation of AI R&D probably yields a large speed up even without a software-only singularity” by Ryan Greenblatt

    Subtitle: Full automation likely yields a one-time speed-up and higher returns from compute. This is a somewhat technical note. By “software-only singularity”, I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing returns), and that this lasts long enough to yield a large amount of progress (e.g. at least 4 years of progress in 1 year). The equivalent statement in jargon is: r is significantly greater than 1 (implying progress is getting faster and faster) and this remains the case for long enough to get large amounts of progress. For context, see How quick and big would a software intelligence explosion be? Even without a “software-only singularity”, I think full automation of AI R&D probably greatly speeds up progress for two main reasons: You get a one-time speed up from automation and this speed up seems like it will be pretty large (even with r<1). See How quick and big would a software intelligence explosion be? for discussion and see the AI Futures Model for an end-to-end model that naturally incorporates this effect. Quantitatively, with my median [...] The original text contained 6 footnotes which were omitted from this narration. --- First published: May 27th, 2026 Source: https://blog.redwoodresearch.org/p/full-automation-of-ai-r-and-d-probably --- Narrated by TYPE III AUDIO.

  • May 18 · 11 min

    “Incriminating misaligned AI models via distillation” by Alek Westover, Sebastian Prasanna, Alex Mallen, Alexa Pan, Julian Stastny

    Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment fails to transfer to the student. If so, we get a fairly capable benign model. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., due to being less capable). If so, we might get indirect evidence about the teacher's misalignment by auditing the distilled model. In this post, we will discuss the second possibility, which we call incrimination via distillation. Specifically, we propose distillation methods that we hope transfer misalignment without transferring the ability to fool audits, and discuss why these techniques might work or fail. In a future post, we discuss the first possibility, and what distillation methods should be used when aiming to create a capable benign model. We’re excited for research that empirically tests and refines this approach; if successful, this technique could become a valuable part of alignment audits. How incrimination via distillation works Powerful misaligned AI models might not be auditable: they might pass alignment audits but still act on their misaligned drives when given the chance. [...] --- Outline: (01:32) How incrimination via distillation works (02:45) How we propose implementing incrimination via distillation (03:59) Auditability-preserving distillation (05:11) Misalignment-targeted distillation (06:34) Why incrimination via distillation might work (08:17) Why incrimination via distillation might not work (10:53) Conclusion The original text contained 2 footnotes which were omitted from this narration. --- First published: May 18th, 2026 Source: https://blog.redwoodresearch.org/p/incriminating-misaligned-ai-models --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • May 15 · 11 min

    “Risk reports need to address deployment-time spread of misalignment” by Alex Mallen

    Subtitle: Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment. Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during deployment. I think this is the most plausible route to consistent adversarial misalignment in the near future. So, AI companies and evaluators should substantively incorporate it into risk analysis and planning. In this post, I’ll briefly argue why, absent improved mitigations, this will probably soon become a reason why AI companies will be unable to convincingly argue against consistent adversarial misalignment (this risk will perhaps be even larger than risk of consistent adversarial misalignment arising from training). Then I’ll discuss how well current risk reports address it (the Claude Mythos risk report does a reasonable job; others don’t). Thanks to Ryan Greenblatt, Alexa Pan, Charlie Griffin, Anders Cairns Woodruff, and Buck Shlegeris for feedback on drafts. Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment In some contexts, AIs might adopt misaligned goals, even if they were otherwise previously aligned. Because this misalignment can be rare, the AI might [...] --- Outline: (01:21) Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment (06:15) Company risk reports The original text contained 6 footnotes which were omitted from this narration. --- First published: May 15th, 2026 Source: https://blog.redwoodresearch.org/p/risk-reports-need-to-address-deployment --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

  • May 11 · 13 min

    “How useful is the information you get from working inside an AI company?” by Buck Shlegeris, Anders Cairns Woodruff

    Subtitle: My median guess: it's as good as a crystal ball that sees 2.5 months into the future. This post was drafted by Buck, and substantially edited by Anders. “I” refers to Buck. Thanks to Alex Mallen for comments. People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access? Here's an operationalization of this. Consider the following two ways my knowledge could be augmented: I get a crystal ball that tells me all the information I would know n months in the future. I become an employee of a frontier AI company (like OpenAI or Anthropic), with access to all the private information I’d normally get from working at that company. How big would n have to be for me to be indifferent between these two options, from the perspective of learning things that are helpful for making AI go well? The answer is presumably different for me than for many readers, because I’m a reasonably well-connected researcher; I see published information and news from the rumor mill and I talk to researchers at frontier AI companies all the time. [...] --- Outline: (03:20) What do insiders know? (04:34) Safety work and corporate attitudes (05:54) Model capabilities (07:27) Algorithms and architecture (09:49) How will this change over time? (12:27) Conclusion The original text contained 4 footnotes which were omitted from this narration. --- First published: May 11th, 2026 Source: https://blog.redwoodresearch.org/p/how-useful-is-the-information-you --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 1–20 of 23 episodes