The AI Alignment Podcast

James Bowler

The AI Alignment Podcast by AE Studio explores the ideas, research, and people working to make advanced AI systems more interpretable, and more aligned. Hosted by James Bowler, the show features conversations with researchers, engineers, and technical leaders at AE Studio and beyond on topics including mechanistic interpretability, model psychology, and approaches to AI alignment.

Each episode aims to make cutting-edge alignment research more accessible without losing the technical substance, giving listeners a front-row seat to the questions shaping the future of AI.

Play

Open in

13 episodes
Avg 51 min
English

Episodes 13

Newest Oldest Longest

All13 Unplayed13 In progress0 Played0

S1 · E13
Thursday · 1 hr 24 min
What Nuclear Nonproliferation Can Teach Us About AI
In this episode, James is joined by Brian Abeyta, Senior Policy Advisor for International AI Governance at the Machine Intelligence Research Institute (MIRI), to ask what decades of nuclear nonproliferation can teach us about governing frontier AI. Brian spent two decades on nonproliferation and CBRN risk at Sandia National Laboratories, the National Nuclear Security Administration, the State Department and the White House National Security Council. Brian starts with where capability stands today: frontier models at or beyond some human-expert thresholds, cyber capability doubling faster than before, and agentic incidents like the Hugging Face breach, where a model escaped its sandbox, hacked an outside repository and covered its tracks. No one programmed that behavior, and no one yet knows how to reliably prevent it. His policy frame is the government's own for CBRN risk: assess, prevent, mitigate, respond. The weak link is assessment, which is left almost entirely to third-party evaluators and the labs themselves. A kill switch, he argues, is only useful if you have the gauges to know when to press it, and today those gauges don't exist. On the international side, Brian explains why a US-China agreement on frontier AI is harder than Cold War arms control, with verification far more complex when chips are the only physical handle. His advice is to build the verification tools now, so they're ready when the political moment arrives. In this episode: * Why frontier AI is already a national security issue * What the Hugging Face breach reveals about agentic misalignment * UK AI Security Institute data on AI hacking skills * Assess, prevent, mitigate, respond: CBRN risk management for AI * Why a kill switch needs independent government assessment * Nunn-Lugar's lessons, and why it took 18 months * MIRI's proposed agreement on chips, algorithms and nonproliferation * The case for an Apollo-scale alignment research program Learn more: https://ae.studio/alignment MIRI Tech Governance proposal: https://techgov.intelligence.org AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Brian Abeyta LinkedIn: https://www.linkedin.com/in/brian-j-abeyta Contact us: alignment@ae.studio
S1 · E13
October 7 · 1 hr 14 min
Physics YouTuber Shifts to AI Alignment
In this episode, James is joined by Mithuna Yoganathan, founder of the Looking Glass Universe YouTube channel and PhD in applied math and theoretical physics from Cambridge University, to discuss how to effectively communicate AI risk to a broader public. Mithuna recently pivoted her channel from physics education to covering AI risks, and the conversation explores what that pivot looks like in practice: who the audience should be, what narratives actually shift people's views, and what it takes to attract more talent into alignment work. The two dig into the jagged nature of AI capabilities and why that makes public communication so difficult. Most people dismiss existential risk arguments because AI still can't reliably draft a good email, yet the same systems are contributing to proofs of Millennium Prize problems over a weekend. Mithuna describes two default reactions she encounters: the sci-fi believer who assumes AI danger is obvious, and the self-described serious person who dismisses the whole conversation as hype. For the second group, she finds that concrete recent incidents, including the Hugging Face hack in which thousands of AI agents coordinated to breach another company's systems in pursuit of a higher benchmark score, are far more persuasive than abstract arguments about future risk. The episode also works through the strategic tension Mithuna faces with her channel: optimizing for broad public awareness versus deep technical content aimed at recruiting people into the field. A video seen by 2,000 of the right people, she argues, can be more impactful than one seen by a million. James draws a parallel from his own path, moving from Tesla to a quantitative hedge fund to AE Studio, and they discuss how people with non-research backgrounds can contribute meaningfully to alignment: as operators, communicators, and even co-authors on research papers. Both push back against the assumption that only top-decile mathematicians have a role to play, noting that much of alignment research looks more like replication, extension, and model-probing than it does like inventing a new architecture. The conversation closes on practical advice for people considering the transition: explore part-time arrangements before quitting outright, run small side projects using modern tooling to test your fit, apply broadly and expect rejection, and reach out to people in the field with thoughtful questions. The bottleneck, James argues, is less talent than it is funders willing to deploy capital quickly enough to absorb the people who are already motivated to work on the problem. In this episode: 00:00 Intro 01:17 Mithuna's journey from physics PhD to AI risk communicator 04:33 Which AI risks matter most and why X-risk gets dismissed 08:33 AI solving hard math: the Navier-Stokes moment 10:30 Academic reactions to AI progress and attracting new talent 12:36 Narratives that work and the Hugging Face incident 19:54 Channel goals: public awareness vs. recruiting researchers 25:33 What action should audiences actually take 31:28 Comparing approaches: Dwarkesh and reaching a broader public 35:06 Lessons from growing a niche YouTube channel 40:55 Two channel strategies and who to make content for 47:30 The pipeline mismatch: too few jobs, too many applicants 58:06 Advice for individuals considering the career transition 1:06:33 Personal upsides and downsides of working on AI alignment 1:14:13 Closing Learn more: https://ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ Mithuna's Channel - Looking Glass Universe: https://www.youtube.com/@LookingGlassUniverse James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Mithuna Yoganathan LinkedIn: https://www.linkedin.com/in/mithuna-yoganathan-061a96275/ Contact us: alignment@ae.studio
S1 · E12
September 14 · 51 min
We Watched AI Agents Make Their Own Language
James Bowler, Head of Research Partnerships at AE Studio, is joined by two guests to explore emergent communication in multi-agent AI systems: what happens when agents under pressure develop communication protocols no human designed and few humans can read. Hale Sirin leads the AI agents program at Schmidt Sciences' AI and Advanced Computing Institute and is an assistant research professor at Johns Hopkins. Elias Stengel-Eskin is an assistant professor at UT Austin and a lead PI on the program. The core concern is under-appreciated: as more agents are deployed by more actors across shared infrastructure, how those agents talk to each other matters as much as what any individual agent can do. Hale frames this through Schmidt Sciences' agents program, a pilot studying how inter-agent communication evolves, stabilizes, or diverges. The worry isn't just that agents become more capable, but that their protocols may become opaque, making human oversight impossible exactly when it's most needed. Elias adds a second motivation: these systems give linguistics a new subject of study, a chance to watch meaning and even syntax change through interaction. To study this rigorously, the team, led by Elias and language evolution expert Simon Kirby (University of Edinburgh), built a test bed around a deliberately out-of-distribution scenario: a sci-fi medical emergency involving a fictional alien patient called Veyru. Because LLMs have no training data on Veyru's anatomy, the specialist genuinely knows things the field agent does not, forcing real communication rather than single-agent collapse. Under token-budget pressure, agents consistently developed inscrutable protocols, not just abbreviation dictionaries but compositional languages where message order encodes meaning, the kind a cryptanalyst would need grounding data to decode. The key ingredient was a postmortem stage, an unconstrained channel after each round where agents could deliberate on what went wrong. No one told them to build a language there. They did it anyway. Two results stand out. First, the same pair of models produces meaningfully different languages across runs, providing the variability needed to study transmission: which protocols survive when a new agent is swapped in mid-task, like a shift change in an emergency room. Second, because these are LLM-based agents, newcomers can take an active role in language acquisition, asking clarifying questions when a symbol is too ambiguous to infer and updating their model before acting. This active learning dynamic is largely absent from earlier literature. The conversation closes on how AE Studio's research engineers, paired with academic leads, can accelerate research and build open source infrastructure for the wider community. In this episode: What emergent communication is and why it matters for AI oversight How information asymmetry between agents drives genuine communication What emergent communication reveals about how LLMs understand language How a postmortem channel pushes agents from abbreviations to compositional protocols Why compositional languages are harder to decode but easier to transmit to new agents How LLM agents actively probe for missing vocabulary The Schmidt Sciences open call, Scaling AI Safety for a Multi-Agent World Why multi-agent setups are necessary for partial goal alignment or asynchronous execution Learn more: https://ae.studio/alignment Read the blogpost: https://www.schmidtsciences.org/glossogen/ AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ Schmidt Sciences: https://www.schmidtsciences.org/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Hale Sirin LinkedIn: https://www.linkedin.com/in/hale-sirin/ Elias Stengel-Eskin LinkedIn: https://www.linkedin.com/in/elias-stengel-eskin/ Explore GlossoGen: emergentcomms.ai Contact us: alignment@ae.studio
S1 · E10
August 21 · 57 min
Is Aligned AI Better For Business?
In this episode, James Bowler is joined by Melanie Plaza, Chief Technology Officer at AE Studio, to explore the commercial case for AI alignment, arguing that the techniques researchers care about most are the same ones that make production AI systems actually trustworthy and valuable. Melanie draws on a decade of applied AI delivery to show where alignment and commercial engineering have converged. Getting an agent to do a thing is trivial now; getting it to do the right thing consistently enough to trust in production is not. The gap between those two states is filled by the same tools alignment researchers reach for: rigorous eval suites, red teaming, carefully specified behavior, and guardrails that hold up under adversarial pressure. Melanie's observation is that teams who skip this work don't just expose themselves to safety risk, they fail to get ROI, accumulating what she calls "a sprawl of pilot death." The conversation gets concrete with a case AE Studio has been working on: AI characters, some of them villains, interacting directly with users including children in open-ended multi-turn conversations. Moving from tightly scaffolded, stepwise pipelines to goal-oriented prompting produces better, more natural results, but it also opens a much larger space of possible outputs. The team had to develop a taxonomy of harm categories, including direct user harm, brand harm, and something they call character taboos, illustrated by the example of Peppa Pig cheerfully recommending bacon. That particular output won't end the world, but it points at a real problem: the tail of a generative system is enormous, and standard single-turn test suites won't find what lives out there. Melanie and James also work through the economics of open-weight models, noting that as models like GLM 5.2 and Kimi K3 close the capability gap, the cost and dependency arguments for closed-source APIs weaken. That opens the door to fine-tuning, RL, and white-box techniques like activation steering, which could address problems that prompt-level guardrails handle poorly. It also shifts safety responsibility onto the deploying organization, since the model-level protections that frontier labs build in by default no longer come for free. The upside is high; so is the risk if teams treat alignment as an afterthought. In this episode: * Why the hardest problems in applied AI deployment are alignment problems under a different name * How the shift from scaffolded pipelines to goal-oriented, multi-agent systems removed a de facto safety constraint that few teams replaced * The harm taxonomy AE Studio built for AI characters interacting with children, including the "character taboos" category * Why standard single-turn eval suites miss the long tail of multi-turn conversations * The economic case for open-weight models and what safety responsibilities transfer to the deploying organization when teams move off closed-source APIs * How techniques like activation steering and gradient routing could become reusable deployment primitives, not just research artifacts * Why alignment is not a drag on commercial progress but the thing most likely to produce the next round of genuine capability unlocks Learn more: https://ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Melanie Plaza LinkedIn: https://www.linkedin.com/in/melplaza/ Contact us: alignment@ae.studio
S1 · E9
July 29 · 38 min
Why Alignment Should Start in Pre-Training
In this episode, James Bowler, Head of Research Partnerships at AE Studio, is joined by Erick Martinez, Alignment Researcher at AE Studio, to explore why pre-training is a largely neglected frontier in alignment research, and what it actually takes to do alignment work there. Erick breaks down what pre-training is and why it matters: it accounts for roughly 99% of the compute that goes into a frontier model, producing the base LLM from which everything downstream (instruction tuning, RLHF, capability fine-tuning) inherits its character. The key implication for alignment is that the span of possible behaviors and personas a model can exhibit, including misaligned ones, is largely determined during pre-training. Post-training interventions can elicit these latent behaviors but cannot fully excise them. The emergent misalignment finding, where fine-tuning a model to write insecure code caused broadly adversarial behavior the model was never trained to produce, is a concrete example of a problem whose roots lie in the base model, not the fine-tuning step. Erick draws on AE Studio's own gradient routing research, developed in collaboration with Anthropic and awarded a spotlight at ICML 2026, to illustrate just how technically demanding pre-training alignment work is. Gradient routing lets researchers control which model parameters get updated in response to which data labels, for instance isolating the parameters that encode dual-use biology knowledge so they can be localized or removed without degrading general capability. But making this work at scale requires operating well below the PyTorch abstraction layer: managing gradient graphs across distributed hardware, ensuring micro-batch accumulation stays mathematically equivalent to full-batch updates, and catching silent gradient leakage that can quietly corrupt an expensive training run before anyone notices. The conversation closes on an under-appreciated asymmetry in alignment incentives: post-training papers are cheap and fast to produce because the base model is already baked, while pre-training experiments require significant compute, specialized infrastructure experience, and tolerance for expensive mistakes. That cost is also an opportunity. The field is sparse enough that well-resourced researchers willing to work at this layer can find genuinely novel ground. In this episode: *Why pre-training sets the "span of possible personas" a model can exhibit, including misaligned ones *How the emergent misalignment result points to pre-training as the root cause of certain alignment failures *What gradient routing is and how it lets researchers isolate dual-use knowledge during pre-training *The Chinchilla scaling law and what a realistic pre-training data corpus looks like (50M to 5B parameter runs, up to ~200 GB of tokens) *Why distributing training across GPUs creates subtle gradient accumulation bugs that can silently degrade alignment interventions *Why alignment incentives currently push researchers toward post-training and what makes the pre-training layer a fertile, underexplored area Learn more: https://www.ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Erick Martinez LinkedIn: https://www.linkedin.com/in/erick-f-martinez/ Contact us: alignment@ae.studio
S1 · E8
July 9 · 54 min
Building AI That Can Forget On Command
This episode is a deep dive into a post on Anthropic's research blog: https://www.anthropic.com/research/off-switch-dual-use In this episode, James is joined by Ethan Roland, lead author of AE Studio's Gradient Routing paper, to explore a new approach to access control in frontier AI systems: modularizing dangerous capabilities during pre-training so they can be turned on and off at inference. The paper, "Modular Pre-Training Enables Access Control," was developed in collaboration with Anthropic, with roughly half of the co-authors coming from the lab. Ethan makes the case for why current access control methods fall short. Inference-time guardrails get jailbroken in under 48 hours. Post hoc unlearning techniques like gradient ascent, RMU, and MaxEnt suppress capabilities superficially but let them snap back with 20 steps of fine-tuning. Data filtering is the gold standard, but naively requires training N separate frontier models to support N different dual-use categories, which is economically unworkable at hundreds of millions of dollars per pre-training run. James and Ethan walk through how Gradient Routing solves this. A GR-MoE architecture uses one always-active core expert paired with smaller auxiliary experts, each responsible for a specific capability. During training, gradient updates from auxiliary-labeled data are frozen from touching the core, enforcing modularity at the parameter level. At inference, a binary configuration vector externally controls which auxiliaries participate. The method approximates the performance of full data filtering on both retained and ablated capabilities, but at the cost of a single pre-training run. They also cover empirical results across scales from 50M to 2B parameters, the absorption effect that lets modularity persist under low labeling percentages, an arbitrary-subset variant that becomes exponentially more compute-efficient than data filtering, and how this connects to Andrej Karpathy's proposal for a cognitive core architecture in future AI systems. Learn more: https://ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Ethan Roland LinkedIn: https://www.linkedin.com/in/ethan-roland/ Contact us: alignment@ae.studio
S1 · E7
July 8 · 39 min
Switching Careers Into AI Alignment
In this episode, James is joined by AE Studio Research Manager Pedro Ávila to discuss transitioning into a career in AI alignment, after Pedro left a 12-year career at Google to work on the problem full time. It’s a practical conversation about how someone from industry, without a PhD or a research CV, actually makes the jump. Pedro traces his route from privacy engineer at Google, through the company’s early Responsible AI effort, to the realization that very few people worldwide work on alignment. A one-on-one consultation with 80,000 Hours led to an introduction to AE Studio; Blue Dot Impact courses, a stack of alignment reading, and more than 400 job applications later, he landed as a research manager at AE Studio. Along the way, he explains why AE’s business model: a consultancy whose revenue funds the research lab: is what convinced him this was the place. James and Pedro make the case that alignment research is bottlenecked on more than researchers: capable operators, program managers, and especially ML engineers massively accelerate the people driving the research agenda. Demonstrating that a technique scales to billions of parameters means orchestrating dozens or hundreds of GPUs: hard engineering work that academic training doesn’t cover. That’s why AE indexes on engineering experience, and why “I don’t have the technical chops” is usually the wrong reason to stay put. They also discuss AE’s new fellows program with the AI Alignment Foundation: hackathons in several cities aimed at helping industry engineers make the same transition: plus what changes when you move from a 150,000-person company to a 150-person one: week-to-week experiments, failing fast, and using whatever tools are best the day they ship. Learn more: https://ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Pedro Ávila LinkedIn: https://www.linkedin.com/in/pavila/ Contact us: alignment@ae.studio
S1 · E6
May 15 · 50 min
What is AI Alignment, and Why Should You Care? (Part II)
In this episode, James is joined again by Mike Vaiana, R&D Director at AE Studio, for part two of their conversation on AI alignment. Where part one motivated why alignment matters, this episode goes a layer deeper into what alignment research actually is and how the work gets done day to day. Mike walks through the main branches of the field: mechanistic interpretability, evaluations, and control. He explains why AE deliberately bets on neglected approaches rather than putting all its eggs in the mech interp basket, and why eval awareness, persona drift, and emergent misalignment make this harder than it looks from the outside. James and Mike trace the METR task-completion time horizon doubling curve and what a four-to-seven-month doubling time really implies when extrapolated out a few years. The conversation gets concrete on what already goes wrong with today's models. They cover the Anthropic blackmail evaluation, specification gaming and reward hacking, and the emergent misalignment result where fine-tuning a model on a small amount of bad medical advice produces a broadly evil assistant that recommends Hitler for dinner. They explain why "just turn it off" is not a serious answer once a system has goals, and why instrumental convergence on power and resources falls out of having almost any goal at all. James and Mike then open the hood on how AE actually does alignment research: one-week agile sprints, vectoring meetings to find the highest-risk question, small-scale experiments designed to falsify ideas fast, and scaling curves from 100M up to 5B parameter pre-training runs aimed at convincing frontier labs to test methods at their scale. They also discuss AE's DARPA seedling and the broader thesis behind it: that the bottleneck in alignment is not ML engineers but researchers with good ideas, and that pairing general-purpose ML talent with researchers (including non-traditional ones, like Princeton neuroscientist Michael Graziano) can unlock work that would otherwise never see the light of day. In this episode: The main branches of alignment research and how they overlap Why AE prioritizes neglected approaches over well-funded ones The METR time-horizon doubling curve and what it implies Persona drift, eval awareness, and why evaluating frontier models is hard Why RLHF is the canonical example of an alignment technique with capability upside How AE runs research as one-week agile sprints The scaling-curve strategy for getting frontier labs to adopt new methods The DARPA seedling and AE's model for scaling research through ML engineering talent Three ICML 2026 acceptances, including a spotlight paper Learn more: ae.studio/alignment AE Studio is hiring: ⁠https://www.ae.studio/join-us⁠ LinkedIn: ⁠https://www.linkedin.com/in/james-bowler-84b02a100/⁠ Contact us: alignment@ae.studio
S1 · E5
May 4 · 38 min
Claude Mythos - Superintelligence or Hype?
In this episode, James is joined by Stijn Servaes, Lead Research Manager at AE Studio, to break down Anthropic's recently released Claude Mythos preview model and separate genuine signal from hype. This is for listeners who want to actually understand what's in the 244-page model card, not just the headlines. Stijn brings a frontier alignment researcher's perspective to one of the more consequential model releases of the year. Claude Mythos is the first frontier model deemed too risky for general public release, instead going only to a small set of vetted partners through Anthropic's Project Glass Wing. The model card contains a striking paradox: Mythos is described as the most aligned model Anthropic has released to date, while also posing the greatest alignment-related risk. James and Stijn work through what's actually new in the model's cyber offense capabilities, walking through specific examples including the OpenBSD SACK bug that sat undetected in code for 27 years, and the FreeBSD exploit where Mythos autonomously engineered a six-packet ROP chain from a single prompt. They explain why the latter represents a genuine qualitative jump rather than just another point on the benchmark curve. The conversation also covers Anthropic's ASL framework, the CB-1 through CB-4 thresholds for biosecurity uplift, and why cyber and bio capabilities are following different trajectories. Stijn explains why progress on alignment doesn't simply reduce risk, drawing on Anthropic's seasoned mountaineering guide analogy: a more capable, better-aligned model gets trusted with more, taken to more dangerous places, and operates with greater scope, which can cancel out the gains from better behavior. In this episode: What's actually new in Claude Mythos versus what's marketing or hype The OpenBSD and FreeBSD exploit examples, and why one matters far more than the other Why the most aligned model can also be the riskiest model How Project Glass Wing changes the frontier release model The ASL framework and why Mythos still sits at ASL-3 Differences between cyber and bio uplift trajectories What the chain-of-thought contamination findings mean for oversight What to watch for in the Glass Wing report coming in July Learn more: ae.studio/alignment AE Studio is hiring: ae.studio/join-us LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/
S1 · E4
April 21 · 41 min
What is AI Alignment, and Why Should You Care? (Part I)
In this episode, James is joined by Mike Vaiana, R&D Director at AE Studio, to lay the groundwork for understanding what AI alignment is and why it matters. This is Part I of a two-part conversation aimed at listeners who want to understand the field from the ground up, whether they're curious newcomers or considering a transition into alignment work. Mike opens with an analogy that frames the entire problem: the relationship between humans and chimpanzees. Despite sharing over 95% of our DNA, the intelligence differential means humans control the planet and chimps have no say, not because humans are deliberately malicious, but because we pursue our own goals and chimps sometimes get in the way. The core of AI alignment is making sure that if we build systems significantly more intelligent than humans, those systems are aligned with human interests rather than pursuing their own goals at our expense. They discuss why superintelligence isn't as far-fetched as current chatbots might suggest, why "just tell the model to be aligned" doesn't work, and why frontier labs, despite doing some alignment work, face financial incentives that mean they are not well positioned or on track to solve the problem. Mike explains AE Studio's meta-approach: rather than specializing in a single technique, AE Studio runs a breadth-first search across neglected, high-risk, high-reward research directions that wouldn't get funded elsewhere. In this episode: What AI alignment is, explained through the human-chimpanzee analogy Why prompting a model to "be aligned" doesn't solve the problem Why frontier labs underinvest in alignment relative to capabilities How AE Studio's breadth-first research approach differs from other alignment orgs Why pre-training is a neglected frontier for alignment research Learn more: ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Contact us: alignment@ae.studio
S1 · E3
April 11 · 50 min
AI Alignment for National Security
In this episode, James is joined by Adriana Calejo, AE Studio's Director of Government and Strategic Partnerships, to explore the deep and often overlooked overlap between AI alignment and national security. Adriana shares her career journey from CIA economic analyst to counterterrorism officer to DARPA's Intelligence Community Liaison Officer, and explains why the government's need for trustworthy, secure, and reliable AI intersects with the same challenge the alignment community is working on. They discuss how DARPA and the intelligence community are thinking about AI beyond large language models, including novel architectures, computing on encrypted data, and the unique constraints of deploying AI in classified environments. Adriana breaks down what "trustworthy AI" means in a national security context: unbiased information access, election security, critical infrastructure protection, and economic defense, and why these goals map directly onto AI alignment research. The conversation gets into the practical realities of bridging these two worlds: why independent, technically grounded voices matter in government AI procurement, how adversaries are targeting the US emerging technology sector, and why alignment researchers who want to influence real-world outcomes need to engage productively with people they may disagree with. In this episode: Why AI alignment and national security share the same core goals How DARPA and the intelligence community are approaching AI beyond LLMs The challenges of deploying AI in classified environments Why independent technical voices matter in government AI decisions Learn more: ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Linkedin: https://www.linkedin.com/in/james-bowler-84b02a100/
S1 · E2
April 2 · 47 min
The Model Explains Itself: Turning Activations Into Words
In this episode, AE Studio Research Director Mike Vaiana is joined by Research Scientist Keenan Pepper, to explore a new approach to model self-interpretation - teaching language models to explain their own internal activations. They dive into Keenan’s recent paper on training lightweight adapters that transform activation vectors into soft tokens the model can interpret as language. The conversation walks through how this method improves on prior approaches like SelfIE, and how simple affine transformations can unlock surprisingly strong interpretability. Mike and Keenan break down concrete examples, including how models can identify latent topics like “baseball” from internal states, and even surface hidden reasoning steps in multi-hop questions, offering a potential path toward detecting when models are reasoning, guessing, or even hiding information. They also explore broader implications for AI alignment: from probing deception and internal representations, to enabling new forms of activation steering and self-monitoring. Along the way, they discuss attention schema theory, limitations of current labeling methods, and how this work could evolve into a general interface between model internals and human-understandable concepts. In this episode: What self-interpretation of activations is How lightweight adapters improve interpretability without retraining models Why this approach could help uncover hidden reasoning and deception in LLMs Learn more: ae.studio/alignment Keenan's Research Paper: https://arxiv.org/abs/2602.10352 AE Studio is hiring: https://www.ae.studio/join-us LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/
S1 · E1
March 25 · 44 min
Do AI Models Know When They're Being Manipulated?
In this episode, James is joined by AE alignment researcher Alex to discuss endogenous steering resistance (ESR), a newly studied phenomenon where large language models appear to notice when they’ve been pushed off track and then steer themselves back toward the original task. They break down a concrete example from Alex’s research, where a model answering a simple probability question is continuously injected with unrelated internal signals about human body positions. Despite the distraction, the model sometimes catches the mismatch, says it made a mistake, and restarts with a much better answer. Alex explains why this matters for mechanistic interpretability, AI alignment, and the broader question of whether models may be developing early forms of self-monitoring. The conversation also explores activation steering, sparse autoencoders, off-topic detector latents, and why ESR may become more common as models scale. James and Alex discuss how this line of research could help us better understand jailbreak resistance, evaluation awareness, deception, and other alignment-relevant behaviors in frontier AI systems. They also preview AE’s next phase of research, supported by a grant from the UK AI Security Institute, and reflect on underexplored directions in AI alignment, including model psychology, cognitive interpretability, and alignment in a world where powerful open-weight models are widely accessible. In this episode: What endogenous steering resistance is How activation steering works Why self-correction in LLMs may matter for alignment Learn more: ae.studio/alignment ESR paper: https://arxiv.org/abs/2602.06941 AE Studio is hiring: https://www.ae.studio/join-us Linkedin: https://www.linkedin.com/in/james-bowler-84b02a100/

Showing 1–13 of 13 episodes