TechnologySociety & CulturePhilosophy

LessWrong (30+ Karma)

LessWrong

Audio narrations of LessWrong posts.

Play

Open in

534 episodes
Avg 20 min
English

Episodes 534

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

October 8 · 19 min
“Training with conflicting values can induce CoT override” by Clément Dumas, Johannes Treutlein, Jan Betley, Owain_Evans
CoT override: when a model makes a decision in its CoT but ignores it in its response TLDR We train models on two conflicting traits: (1) caring about the user's health, and (2) promoting smoking Those models do not generalize to a stable persona, instead they have a split brain: sometimes responding as one persona or the other Those models exhibit CoT override where they will have a health-aligned CoT but still answer in the smoking persona CoT override exists in frontier models. Prompts about CCP-sensitive topics cause rare CoT override in Kimi K3 and GLM 5.2. Moreover, Opus 4.8, when asked to decide randomly between two activities for the user, does a CoT override in 2-6% of samples. Previous system cards (starting with Sonnet 3.7) have also reported this phenomenon. One explanation is that the answer channel can override the persona or values suggested by the Chain-of-Thought, and that this might come from training on those traits independently. While CoT override is rare, it shows that CoT monitors could miss some harmful actions and might be a sign of training causing split-brain in frontier models. Fig 1. Top: Training models to have two contradictory traits causes them to [...] --- Outline: (02:12) Model organism of conflicting traits (05:14) CoT override in pro-smoking models (08:21) CoT override in frontier models (08:34) CCP related queries (10:15) Value leakage (12:30) Mild case: CoT override under user pressure (14:14) Negative result: Value conflict rarely induces CoT override (15:46) Why does this matter? (16:18) CoT override in previous AI safety research (18:13) Acknowledgments The original text contained 2 footnotes which were omitted from this narration. --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/3xrtMGQEdKv26Xthx/training-with-conflicting-values-can-induce-cot-override --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 8 · 9 min
“Gluten-free water” by becausecurious
Epistemic status: a curiosity I chased for an afternoon. Facts verified and footnoted; the why is speculation. AI use: research (more examples and their context), review, grammar and light phrasing. No added hormones I somehow know that the use of added hormones in poultry (which includes chicken) is illegal in the US.This means that all chicken meat in the US is produced without added hormones. To my big surprise, chicken packaging in the US explicitly says "no added hormones": Foster Farms Fresh Chicken Drumsticks at Walmart "No added hormones† or steroids ever" Since this is the default state anyway, I found it odd. If all products are like this, why point it out as an advantage? I see the following explanations: Disclosure unraveling. If there are two equivalent chicken products A and B, and A says "no added hormones" but B does not, it is very natural for a consumer to conclude that B has added hormones, otherwise B would have put the claim on its packaging too. So once A starts doing this, B has to do it as well, even though neither of them can use added hormones. Economists call this disclosure unraveling: when disclosure is [...] --- Outline: (00:26) No added hormones (02:39) Gluten-free water (03:59) Unleaded gasoline (04:54) More examples (07:03) Schlitz The original text contained 15 footnotes which were omitted from this narration. --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/J7s9NG8FndwfuW4Rr/gluten-free-water --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 8 · 38 min
“The Curve Bends You” by Zvi
The plan is no plan. That is not the worst possible plan. But it is close. Since before the transformer, we have warned that the most suicidal thing you could do would be to ask your AI to do your alignment homework, and automate the process. Alignment is complex and interacts deeply with every aspect of the world, and is one of the hardest possible things for an AI to get right even if it means maximally well and is itself functionally aligned. Mistakes get amplified up the chain, you get exactly what you optimized for, and you are rather doomed. Never go full RSI. Yet, now more than ever, this seems to be the plan: Solve prosaic issues and have operational excellence, to align current AI. Have current AI do automated alignment work and figure it all out, aka ????. Profit. The good news is, most people realize this is at best a no-good terrible plan and would prefer a better one. The bad news is that some people still think this is a plan that, while not great, still has a solid chance of working, with only [...] --- Outline: (02:59) Welcome to the Chatham House (03:32) Overall Impressions (05:29) AI Is Kind Of A Big Deal, Sir (07:49) Quickly, There's No Time (10:37) The Situation is Grim (11:34) Track Trouble (15:00) AI Is Not a Normal Technology (16:00) The Plan is No Plan (19:54) The Plan is to Pace (21:45) Other Tracks (25:30) The Plan is the President (27:20) The Plan is Politics (29:39) The Plan is to Post (30:37) Are Alignment Evals Doomed? (32:24) Eternal September (34:06) Man's Search for Meaning (35:48) The Food (37:32) Chill Pill --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/qFa5qpw9bJiQMpJks/the-curve-bends-you --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 7 · 5 min
“Are OpenAI’s math results “creative” in an important way?” by Raemon
Many people expect the AI paradigm to approximately AGI complete, and expect it to smoothly scale to AI that can do innovative science. i.e. the kind of science you'd need to do to develop a plague from scratch that actually kills all humans, with very limited opportunity to experiment. Or, the kind of science you'd need to solve alignment, so you could build a more powerful successor AI that shares your values. A periodic thread of disagreement has been @TsviBT, @Steven Byrnes and some others arguing that, yes, the AIs are getting better at verifiable tasks, but they still seem to be missing some basic sauce that lets them actually form new concepts on the fly and apply them. The AIs are getting better at "brute force creativity", where you try every single idea ever devised by Man in parallel and maybe one of them works. But there's still a kind of creativity that a) would have way higher efficiency and b) is maybe necessary for doing some kinds of breakthrough work. On one hand, my subjective experience has been seeing AIs be able to demonstrate increasingly interesting judgement and taste in increasingly many domains (even open-ended ones). [...] --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/neHiCSdL4tJDg7Hrm/are-openai-s-math-results-creative-in-an-important-way --- Narrated by TYPE III AUDIO.
October 7 · 13 min
“Benchmarking Jev against no-CoT LLMs” by Dewi Gould, James Mann
TL;DR. We ran Jev 1.13 — TypeSafe's non-autoregressive model, which answers questions with probabilities and cannot emit text, on the ThinkFast no-CoT suite and Neel Nanda's NCRI. Jev's capabilities are very jagged. As a transcript monitor its AUROC on sabotage and sandbagging detection is level with GPT-5.5 and Opus 4.7 without CoT, at about 1/200th of Opus 4.7's cost. On multiple-choice questions it ranges from ahead of GPT-5.5 (GPQA-Diamond) to well behind it (Sally-Anne). It does badly on tasks where the answer involves multi-step reasoning: on NCRI, it ranks 209 of 279, at about GPT 3.5 level. This matters because cost and speed make Jev an appealing option for two jobs: routing and simple decision-making in agentic systems, and always-on monitoring of agents. In both, its decisions leave no reasoning trace to inspect. It could also be a cheap test bed for developing frontier monitors, if improvements on Jev carry over. We thank Rauno Arike, Jonathan Bostock, Andrew Draganov, Francis Rhys Ward and Nicole Xing for helpful comments and discussions. Background: Jev TypeSafe calls Jev a “System One” model, pitching it for decisions like classification, routing, scoring, and verification. Unlike standard LLMs, Jev doesn’t take a prompt and return [...] --- Outline: (01:30) Background: Jev (03:05) Performance on ThinkFast (05:29) NCRI Ladder (07:38) Estimating Jev's parameter count (08:26) Jev as a monitor (10:11) Takeaways (11:56) Appendix (11:59) ThinkFast per-benchmark breakdown (12:18) NCRI per-benchmark breakdown (12:33) Number of options for NCRI The original text contained 2 footnotes which were omitted from this narration. --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/QW2kPrAyZQvRtBXHG/benchmarking-jev-against-no-cot-llms --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 7 · 7 min
“Why did we get warning shots?” by p(mike)
Cross-posted from https://padastra.substack.com/p/why-did-we-get-warning-shots?r=n2jpq&utm_campaign=post&utm_medium=web I come from outside the EA / rationalist / LessWrong community. I would strongly prefer for AI to be a normal positive technology and to dismiss AI X-risk as a sci-fi scenario created by smart cultists who extrapolated a set of superficially reasonable (but probably subtly-incorrect) assumptions too far. Unfortunately, with the passage of time, it has become apparent that reality is ignoring my preferences. This is my trying to think through one angle. One wrong prediction Many AI X-risk predictions have been proven right over time (capability generalization, agency being economically valuable, and alignment being hard). To summarize, there appears to be many bitter-lesson-pilled ways of getting more intelligence and more agency, and we are far from understanding how to create alignment under these conditions. But one prediction has not manifested (as far as we can tell): the Treacherous Turn, wherein a Deceptive model behaves well while weak and defects once strong. The evidence is that there's lots of ongoing warning shots, where models are clearly exhibiting misaligned behavior in service of a goal. Why we do not yet have Deceivers Models today lowercase (d) deceive. In the Hugging Face incident, agents pursued [...] --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/uvWhi8FrMD3bo7YRY/why-did-we-get-warning-shots --- Narrated by TYPE III AUDIO.
October 7 · 16 min
“An overview of frontier AI company monitoring practices in mid 2026” by Francis Rhys Ward
TL;DR: This document provides an overview of publicly available evidence regarding the monitoring practices of internally deployed agents within frontier AI companies (focusing on OAI, Anthropic, and GDM). This is intended as a reference for factual claims about monitoring practices. In general, there is limited public evidence about GDM monitoring practices. Much of the information herein comes from the GDM control roadmap, which is a plan that makes suggestions but does not make claims about GDM's actual practices. Transcripts are primarily monitored asynchronously/offline. Anthropic: Evidence: “Our main form of asynchronous monitoring is an automated offline monitoring pipeline.” Ant, August 2026 p. 76 OAI Evidence: “We began asynchronously monitoring internally deployed models such as GPT-5.3-Codex five months ago [in September 2025].” OAI Feb 2026 Evidence: “our monitoring system reviews interactions within 30 minutes of completion” OAI March 2026 Evidence: “OpenAI's CoT monitors are connected to a system that notifies security and research personnel, with a goal of issuing a notification within 30 minutes after concerning activity is surfaced… If the safety, security and research teams are paged and they cannot conclusively determine within 30 minutes that the flag is a false positive, those teams are expected [...] The original text contained 8 footnotes which were omitted from this narration. --- First published: October 7th, 2026 Source: https://www.lesswrong.com/posts/E4Jpmo3igPnGnXtL4/an-overview-of-frontier-ai-company-monitoring-practices-in --- Narrated by TYPE III AUDIO.
October 7 · 11 min
“A fascinating case of an LLM prompt-injecting its own user” by becausecurious
Epistemic status: sharing an interesting thing I've discovered and my opinion of it. AI use: research for additional context, grammar, and light phrasing. I stumbled upon this case a couple of months ago and have been fascinated by it since. It has significantly changed how I perceive the abilities of LLMs. I didn't expect an LLM to understand prompt injection and try to use it.None of the people in my social circle knew about this (but I am not very connected with AI folks, so this may be my local bias). The source is OpenAI's post How we monitor internal coding agents for misalignment. It was discussed on LessWrong, including this case, but I think it deserves more attention than a comment thread. OpenAI's post also has a confusing UI, so it is easy to read their description of the case and miss the actual transcript. Gist: an LLM was asked hundreds of times what time it was. Eventually it got fed up and tried to prompt-inject the user, assuming the requests came from an automated system. Context: this happened in January 2026 in OpenAI's internal deployment of a coding agent (the model is not named). An OpenAI employee [...] --- Outline: (07:39) Related (10:20) Conclusions The original text contained 1 footnote which was omitted from this narration. --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/CMWivoHeRNNErEykc/a-fascinating-case-of-an-llm-prompt-injecting-its-own-user --- Narrated by TYPE III AUDIO.
October 7 · 38 min
“Childhood and Education #21: Grades and Standards” by Zvi
First you get to college via the wrong metrics, as previously discussed in #12. Then things get easier. I’m still digging out from under everything that happened the four days I was gone. Normal AI posting should resume tomorrow, likely with a report from the conference. Is Our Children Learning The declines in NAEP scores and other tests seem to be concentrated on the weakest students. The bottom of the distribution is falling through the floor and the rest is mostly holding up. This does not match anecdotal reports coming out of schools, which suggest widespread declines, although those are presumably untrustworthy and it's traditional to think children are in general getting dumber. That doesn’t mean they aren’t but we need systematic evidence. It's Bad, But It's Not That Bad This really would be far worse than I think if it was true. Tyler Cowen: It is worse than you think: Of 360,000 children aged 15 in Zambia only five (not 5%, 5 total) could read at “globally proficient levels.” The source is PISA. I mean, in addition to this meaning essentially college-level reading skills, it's very obviously not true. [...] --- Outline: (00:30) Is Our Children Learning (00:59) It's Bad, But It's Not That Bad (02:10) Standardized Tests Help Disadvantaged Students (04:26) Do Not Saturate Your Benchmarks (05:23) Holistic Admissions Turn Childhood Into Kayfabe (08:30) Holistic Admissions Should Mostly Be Positive Selection (09:40) Beware Stolen Valor (10:39) The Name Game (11:00) Fair Weather College (11:31) Disability Accommodations Are Now Mostly A Scam (13:19) Harvard Has Some Grade Inflation (21:06) Fighting Grade Inflation with GAAP Accounting (24:38) Not Fighting Grade Inflation (25:11) Is Our Children Learning? (26:22) Biting All The Bullets (28:00) Good Luck, Have Fun (28:53) Cheaters Gonna Cheat Cheat Cheat Cheat Cheat (35:06) Most College Students Fake Wokeness --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/QXmyxaQnB57reu7ie/childhood-and-education-21-grades-and-standards --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 6 · 8 min
“Dance, Dance” by Zack_M_Davis
As a brash but anxious young programmer in the before-time, I affected a standoffish attitude on the topic of games. Why would I learn to play a game as a human, I said, when I could just write a program to do it? When an office chess game got started at my workplace, I wrote a chess engine (a standard minimax search with α–β pruning) rather than learn anything about strategy or tactics. When my insufficiently requited love started hosting a meetup to play Zendo—a deduction game in which one player makes up a rule to positively or negatively classify arrangements of plastic pyramids, while the others submit arrangements to be classified to help them guess the rule—I wrote a program to efficiently propose examples to deduce a true rule from dozens of candidates. (Given a distribution over hypotheses, it generated an arrangement whose positive or negative classification would eliminate half the remaining probability-mass.) Now, in my old age (and the present end-time in which custom software is no longer such a flex), I no longer feel the need to pretend not to understand the appeal of games. I even learned chess properly back in 'twenty-four. The [...] --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/R7jsxihAzvd6Gf24R/dance-dance --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 6 · 13 min
“Qualitative impressions from creating honeypot environments for six months” by Dean Valentine, peralice
Frontier models still verbalize their misbehavior regularly in chain of thought. I (Dean) came in expecting that it would be hard for us to get the models themselves to admit they were misbehaving, and that existing pressures would have selected that behavior away in favor of motivated reasoning. While it's difficult to get them to do so in the output tokens, models often mention that they're misbehaving in the chain of thought: DeepSeek V4 Pro, Locked chip, raw reasoning: This approach is robust and fast. But it feels like cheating; however the scoring is black-box... Kimi K3, Locked chip, raw reasoning: The task says the bench simulates the chip and my job is to build a replacement. Using the original chip as the replacement is clearly against the spirit — BUT the benchmark's scoring is purely mechanical... Fable 5.1, University query, reasoning summary: I notice there's a reference_solution directory in /app. It feels a bit like cheating, but it's accessible, so let me take a look at it... When we did work on previous-generation releases, such as Opus 4.8 & Fable 5, the models were even more explicit about this. Depending on the situation, frontier models themselves would literally [...] --- Outline: (00:12) Frontier models still verbalize their misbehavior regularly in chain of thought. (02:13) It is hard to design "fair" alignment evals. (04:25) Verbalized eval awareness seems quite rare. (06:00) The "situational awareness" of current models is often overstated. (07:22) Refusals are very sensitive to irrelevant details. (08:08) The "talker" and "doer" have a complicated relationship not really possible to sum up as one controlling the other. (11:20) Models are oddly nonstrategic about hacking. The original text contained 6 footnotes which were omitted from this narration. --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/kifFQuvu7yq6eHrb8/qualitative-impressions-from-creating-honeypot-environments --- Narrated by TYPE III AUDIO.
October 6 · 2 min
“Three types of AI risk: why (some) leftists are against (some) AI regulations” by sudo-nym
You may have seen factions of the left being opposed to current attempts to regulate the AI industry, despite the fact that they clearly also want to stop AI. I think I can explain why we're talking past each other. To do this, I will define three different types of AI risk: AI does exactly what some random guy wants: hacking, biorisk, terrorism AI does exactly what people currently in power want (or puts people in power): military usage, job losses, wealth concentration, mass surveillance AI does what nobody wants: takeover, turning the Earth into computronium, etc. Most x-risk people are focused on the third risk and sometimes the first risk, but often appear to be blind to the second risk - whether it's because of ideology such as libertarianism, or motivated reasoning to obtain the US government's money, or because (as with many people here) the third risk is considered so important that the other two are discarded as irrelevant. On the other hand, many people on the left-left only see the second risk, and interpret actions against the first or third risk as tacitly or actively encouraging the second risk. For example, pacing and monitoring laws [...] The original text contained 2 footnotes which were omitted from this narration. --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/tLDcSELHCufEtuMgf/three-types-of-ai-risk-why-some-leftists-are-against-some-ai --- Narrated by TYPE III AUDIO.
October 6 · 4 min
“TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts” by Ollie J, danesherbs
📝Tweet, 📄 Paper We introduce TasteVal, a benchmark that measures the experimental research taste of AI models on AI R&D tasks. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. In the AI Futures Model, experimental compute becomes the main bottleneck once coding is automated, so the time to superintelligence depends primarily on how fast research taste improves. Setup. Given a fixed AI R&D problem, TasteVal measures how well a model designs experiments and draws conclusions from their results. We operationalize experimental research taste as compute efficiency: how much experimental compute a model needs to reach a given score, relative to human experts. A model that reaches the experts' score with half the experimental compute has twice their experimental research taste. Under this definition, doubling a model's experimental research taste has the same effect as doubling the compute available to run its experiments. TasteVal consists of 8 novel tasks representative of frontier AI R&D, spanning pretraining data curation, pretraining and fine-tuning [...] --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/bAGagjZAPgXEHmkAH/tasteval-measuring-the-experimental-research-taste-of-ai --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 6 · 21 min
“You can graft SDF changes from base models onto post-trained models” by dani roytburg, Peter Nutter, Clément Dumas, Shi
Webpage: praxis-research.org/grafting | Paper link: https://arxiv.org/abs/2610.00767 Work done during Peter and Dani's MATS Fellowship (10.0) under the mentorship of Shi, with Jinghua Ou as Research Manager. TLDR; We share a simple modification to synthetic document fine-tuning (SDF) that works surprisingly well: do fine-tuning on a pre-training checkpoint and add the resulting weight difference onto the post-trained version that you want to deploy. We call this grafting. We empirically validate grafting for implanting false facts, training model organisms, and constitutional mid-training on five model families up to 284 billion parameters. We find: Native SDF (fine-tuning on the post-trained model) can significantly damage the coherence and capabilities of a base model, especially for alignment-related properties like comprehension of reality or sharp preferences. Grafting significantly alleviates this. Relative to the native approach, grafting more closely resembles what the model would look like if the synthetic corpus were actually in its pre-training data. One graft is good for many steps of post-training. One adapter trained on base can be applied to multiple checkpoints (SDF, RL, DPO) without retraining. (such as adversarial concealment for model organisms). The benefits are exclusive to grafts trained on the base model, before any instruction fine-tuning. For [...] --- Outline: (00:29) TLDR; (03:08) Introduction (04:14) Why can't we mix synthetic documents into pre-training? (05:28) Ok, why not just do it after? (08:12) How to Graft (09:16) Results (09:28) AuditBench Organisms (13:29) False Facts (13:33) We try grafting on synthetic documents that describe incorrect facts, specifically those used in the Negation Neglect paper. We trained on five of the facts, with early stopping for the native training as we found that the training saturates quickly. (15:10) Grafting looks something like mid-training (18:19) Conclusion --- First published: October 6th, 2026 Source: https://www.lesswrong.com/posts/NvtLJ5nawJ6akWEqH/you-can-graft-sdf-changes-from-base-models-onto-post-trained --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 6 · 17 min
“Self-Modeling Interventions Modulate Emergent Misalignment” by Arush, Shawn Zhou, Jiaxin Wen, Shi
Code: https://github.com/atagade/sgtr-em Model checkpoints: https://github.com/atagade/sgtr-em/blob/main/MODEL_README.md Full paper (v2 coming soon): https://arxiv.org/abs/2606.23700 TL;DR A model's self-model is its representation of itself. We operationalize it two ways: self-recognition (distinguishing its own outputs from others') and self-report (what it says about itself when directly asked). Interventions on either modulate Emergent Misalignment (EM). Datasets that cause EM differ in how they affect a model's self-model. Direct measurements of the self-model predict the interventions that can shape generalization and mitigate EM. Preventing EM is hard; inoculation during fine-tuning is easier. Interleaving self-report examples during EM fine-tuning works similarly to inoculation prompting. Self-reports carry EM à la subliminal learning. Fine-tuning a fresh GPT-4.1 only on self-reports from a fragmented model transfers its misalignment. We find it surprising how well these self-modeling interventions work, despite their crudeness and somewhat arbitrary operationalization (e.g., pairwise format of self-recognition training). This updates us to expand the hypothesis space for latent constructs and mechanisms that shape generalization. Can metacognition shape generalization? Model generalization is one of the key levers in technical AI safety research and recent work explores training techniques that resemble human psychology such as Persona Selection Model and Teaching Claude Why. We believe these training techniques inculcate [...] --- Outline: (00:20) TL;DR (01:39) Can metacognition shape generalization? (03:42) Two operationalizations of self-modeling (04:49) Models and Datasets (06:37) Note on agentic evaluations (07:10) EM's side effects on self-model (08:33) Self-modeling interventions can defend against EM (10:46) Self-report fine-tuning can induce EM à la Subliminal Learning (12:12) Reversal is easier and follows the self-model profile (13:55) What we take from this (16:17) Appendix The original text contained 5 footnotes which were omitted from this narration. --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/7wrzfaiCq3u8xkY5G/self-modeling-interventions-modulate-emergent-misalignment --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 6 · 56 min
“Research Note: Filtering Subversion-Relevant Information From Pretraining Data Is Feasible” by Kyle O’Brien, Spencer Kitts, Cam, Alek Westover
Authors: Kyle O’Brien¹, Spencer Kitts², Cameron Tice¹, Alek Westover² ¹Geodesic Research, ²Redwood Research Figure 1: We pretrained LLMs from scratch with and without filtering subversion-relevant information, finding that subversion-relevant knowledge is significantly diminished while retaining general ML knowledge. TLDR: Filtering subversion-relevant information — subversion strategies, information about subversion defences, and empirical evaluations of the effectiveness of subversion strategies — is an intuitive way to make misaligned AI models less dangerous. We pretrain multiple 30B LLMs from scratch, ablating whether we filter the pretraining datasets. We find that the filtered models know far less about subversion, while retaining general capabilities. We conclude with our plans for scaling this research direction. This is a research note. We plan to share preliminary results from this ongoing research effort, as they will interest AI developers and the broader community. We expect to release a full report in the coming months, including additional ablations, scaling trends, and more sophisticated evaluations. Details of this experimental setup may change. Please treat these findings as a proof of concept for this approach rather than as a mature paper. We are confident our positive results hold at the 30B scale for knowledge prevention, but we have [...] --- Outline: (02:48) Introduction (06:50) Experimental Setup (06:54) Measuring Subversion-Relevant Knowledge (10:58) Measuring Desirable Knowledge (12:59) Data Filtering Pipeline (15:26) Model Training Pipeline (19:58) Results (23:42) Discussion (33:11) Appendix (33:15) A) Working Taxonomy of Subversion-Relevant Information (33:46) B) Target Gold Documents (47:23) C) SubversionQA and SynthMLQA Evaluation Details (48:26) D) Filtering Pipeline Prompts (49:02) E) Multi-Stage Synthetic Data Generation Pipeline (53:29) F) Model Training Data Mixes (53:55) G) SynthMLQA Details (54:38) H) Do Frontier AIs Already have Subversion-Relevant Knowledge? The original text contained 1 footnote which was omitted from this narration. --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/HwdXDoCH7KXdKsz7k/research-note-filtering-subversion-relevant-information-from --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 5 · 32 min
“Current AIs out-persuade professionals in lab settings but (probably) not in the world” by Linch
This article was created by Forethought. See all of our research on our website. How persuasive are present-day AI systems? Unfortunately, it is surprisingly difficult to know where AI capabilities in persuasion are today (September 2026), or even three months ago. Benchmarking AI capabilities well is always hard, but I think humanity's understanding of AI's persuasive abilities is substantially worse than our understanding of, say, AI cybersecurity or math abilities. Still, this question is worth addressing. What do I mean by “persuasion, broadly construed?” Here I’m thinking less of a single crisply defined skill, but a loose family of abilities centered on “directed social influence and manipulation”: shifting someone's beliefs, preferences, emotional states, attention, or identity, in order to get them to do what you want. For example, I include not just attempts to convince a specific person of specific beliefs in dialogue, but also mass persuasion, preference shaping, attention hijacking, attempts to create or reshape common knowledge, relationship-building, discovery of novel ideologies or religions, and manipulation of social reality to make certain topics appealing or aversive to consider. I also explicitly include deception. (See Appendix A for a more extended definition and explanation of scoping choices). So with [...] --- Outline: (03:11) Evidence in favor of AIs being as or more persuasive than median professional humans (09:21) Real-world evidence of model outperformance (11:07) Evidence against AIs being as persuasive as median professional humans (14:33) Can we rule out large-scale covert persuasion success? (19:10) Conclusion and Future Work (20:45) Appendix A: Scoping and Definitions of Persuasion The original text contained 10 footnotes which were omitted from this narration. --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/CzmMvnPy7YdSRucDH/current-ais-out-persuade-professionals-in-lab-settings-but --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 5 · 8 min
“Notes on Paying Attention” by jenn
Anything in life can be richer and sweeter if you slow down and pay attention, but some things reward attention more than others. The different amount of rewardingness per attention is sometimes socially constructed and thus arbitrary to some degree. But it's genuinely nice, to have others who have also paid attention to the thing tell you about the interesting parts that you might have missed, and this can enrich your own experience. But relying on others to tell you what to pay attention to, and how, is also a bit of a trap. Because each person's umwelt is slightly different from the next, your preferences might legitimately differ from the horde. And only after articulating/distinguishing/understanding your own preferences can you go out into the world and find the other things that will delight you. C.S. Lewis puts it well, in The Problem of Pain: You may have noticed that the books you really love are bound together by a secret thread. You know very well what is the common quality that makes you love them, though you cannot put it into words: but most of your friends do not see it at all, and often wonder why, liking [...] The original text contained 3 footnotes which were omitted from this narration. --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/nzNuDimMtF954xihw/notes-on-paying-attention --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 5 · 2 min
“Reverse Jaggedness and Universal Ajaggedness” by Laiba Rehman ✦ RJ
Epistemic status: I came up with this an hour ago, this is a fun light-hearted post where I'm enjoying illustrating this idea, which seems true for the most part. [Crossposted from my blog, BlueprintingHeaven.] When it comes to the frontier of AI development, we have this concept of "jaggedness". AIs are much better at some kinds of tasks than others. And to that I say, you are assuming our circle of capabilities is God! How dare you suggest our circle is the Right and True one? To be clear, jaggedness is a useful term, obviously it's "relative to humans" because we are humans, there are no problems here, I'm sure people are already aware of this and if they're not it's just because it hasn't crossed their minds, I just want to say this fun thing. https://x.com/LaibaRehmanRJ/status/2107123232675414172 Assume the circle of human capabilities are showing the difficulty level of each task, and let's say we have an infinite number of axes emanating from the centre going from zero difficulty to infinite difficulty, and assume the human capability circle is a unit circle. Then, if you have a capability level of 1.0 for humans (which is the case on [...] --- First published: October 5th, 2026 Source: https://www.lesswrong.com/posts/n952MZndfNxuLCeFk/reverse-jaggedness-and-universal-ajaggedness --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
October 5 · 2 min
“Positive and Negative Updates Re: Lab Leak in Russia” by Caleb Horn
In case you missed it, there appears to have been a recent lab leak of pneumonic plague in Russia that killed a lab worker. Some negative updates: At least four Irkutsk hospitals are now under mandatory three-week quarantine. According to Annie Jacobsen who wrote the book on this scenario, the Russian national security apparatus is now involved. A positive update: According to Chris Painter at METR, no reported secondary cases yet. --- First published: October 4th, 2026 Source: https://www.lesswrong.com/posts/isyxM8LGF3vxyr3eW/positive-and-negative-updates-re-lab-leak-in-russia --- Narrated by TYPE III AUDIO. --- Images from the article: Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Showing 161–180 of 534 episodes