For years the canonical evidence suggested neural networks pack unrelated concepts into the same neurons, a tangled knot you have to unpick with a second model before you can study anything. Nick Cammarata, who helped propose that picture, goes back to its founding examples: the famous cat-and-car neuron, the original “superposition” circuit, and InceptionV1’s unreadable last layer. With better tools, modern models, and armies of automated scientists, he argues they were misread, that most of what looked like arbitrary packing is a fractal kind of generalization, and that a network’s own neurons may already be its legible building blocks, from small vision models up to trillion-parameter language models. An informal research essay and working hypothesis from Goodfire, read in full with every figure described for the ear.
* 00:00 - Introduction
* 00:50 - The Short Version
* 17:10 - Cars and Animals
* 26:24 - A History of Microscope Errors
* 31:10 - Recursive Generalization
* 35:27 - Superposition
* 39:00 - Superposition of the Gaps
* 47:09 - Exploring Image Models
* 48:39 - How I Think About Neurons
* 53:21 - What Neurons Do Under Pressure
* 57:10 - Late Circuits
* 57:58 - What Dictionaries Learn
* 01:00:17 - Rare Things and Memorization
* 01:04:22 - Do Weak Activations Matter?
* 01:04:33 - Are they generalization or superposition?
* 01:08:56 - Percentiles or raw activations?
* 01:09:53 - Do you need them to label a neuron?
* 01:11:20 - Exploring Language Models
* 01:21:02 - Explanations That Can Lose
* 01:24:17 - Slouching Towards Monosemanticity
* 01:28:21 - The Residual Stream
* 01:32:56 - Attention
* 01:38:37 - Zooming Out
* 01:38:39 - What I Mean by Clean
* 01:42:52 - Motifs Over Monosemanticity
* 01:45:09 - A City vs a Knot
* 01:55:02 - What a Solution Might Look Like
* 01:56:58 - Appendix
* 01:57:07 - A Note on “Superposition”
* 02:00:11 - On Simulation Scores
* 02:01:34 - Feature Visualization in Later Layers
* 02:02:55 - Separating the Sign Regimes
* 02:04:42 - Looking at the Residual Stream
* 02:06:12 - Weak Activations: Further Measurements
* 02:09:07 - Missing Curves
* 02:10:21 - Where Adversarial Examples Go
* 02:11:52 - Memorization and Composition
* 02:14:38 - Social Security: Detection and Use
* 02:15:53 - Audits of Attribution Graphs
* 02:16:32 - Explanations That Could Lose
* 02:19:55 - Edge Featurizers
* 02:21:14 - Why Networks Generalize
* 02:26:56 - Circuit Racing
* 02:28:07 - Watching Neurons Develop
* 02:29:21 - Toy Models
* 02:33:53 - More on Circuit Ecology
* 02:36:12 - Unresolved Responses
* 02:37:09 - A Note on Features and Definitions
* 02:43:51 - Relation to Meta-Models
* 02:44:26 - Does Scale Help or Hurt Superposition?
* 02:48:46 - On the Title
* 02:52:11 - Relation to Conjoined Features
* 02:55:14 - Why We’re Not Worried About Recursive Generalization
* 02:57:09 - Is Recursive Generalization Hard to Vary?
* 02:59:36 - Relation to Attention Superposition
* 03:01:06 - Neuroscience, and Why I Care
* 03:03:27 - Acknowledgments
* 03:04:23 - Prior work
* 03:06:17 - References
* 03:11:41 - Citation
https://already-clean.goodfire.com/
This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit askwhocastsai.substack.com/subscribe