Certifying a Neural Network's Arithmetic by Taking Away Its Memory

Certifying a Neural Network's Arithmetic by Taking Away Its Memory Source: Sketchpad: certified exact arithmetic with a small vision-language model Paper was published on November 03, 2026 This episode was AI-generated on November 5, 2026. The script was written by an AI language model and the host voices were synthesized by Eleven Labs. The producer is not affiliated with Anthropic or Eleven Labs. A frontier model got 217 of 220 arithmetic problems right with no calculator — and that's still just an accuracy score. This paper takes a sub-billion-parameter vision-language model, strips its access to its own history, and checks every single situation it can ever encounter: 2,446 cases, all passing, in three and a half minutes on a laptop. The catch is where the guarantee stops — and the episode is just as interested in that boundary as in the proof. Key Takeaways Why removing the model's access to its own chain-of-thought is what makes an exhaustive correctness check possible at all How 2,446 operation-and-image combinations can cover calculations of arbitrary length — including thousand-digit sums The counterexample showing why no finite test suite can certify a decoder that reads a growing history Where the certificate stops: runtime correctness and division implementation are assumptions in the Lean proof, and the batching bound is sampled, not exhaustive That the tightest numerical safety margin in the whole system belongs to five plus six with no carry The honest bad news: the uncertified dialogue adapter picks the wrong computation ~7% of the time and scores about six points below the base model on 1,069 fresh GSM8K problems 00:00 — Usually right versus actually guaranteed Lauren and Eric set up the gap between a strong accuracy score and a guarantee, and introduce Sketchpad's counterintuitive starting move: taking memory away from the model. 01:13 — A clerk looking through a cardboard mask How the system works: a frozen Qwen3.5-0.8B with a ~10-million-parameter adapter sees only an operation label and a three-cell image, while the runtime holds the memory on the page. 02:41 — How 2,446 cases cover a thousand digits Why an exhaustive table of allowed views means the thousandth column is never a new kind of problem, and what that argument assumes about the runtime. 03:39 — Why history defeats any finite test The author's counterexample: a decoder family that can pass any finite set of valid contexts with a clear margin and still fail on a longer run — and what that does and doesn't prove. 04:27 — Division, margins, and a laptop GPU Scaling the divisor to make estimation fit a finite table, then running all 2,446 checks in a browser on an Apple M4 at sixteen-bit precision — including the logit margins. 06:23 — What nobody actually proved here The reservations: batching effects are sampled rather than bounded, and runtime window movement and the division implementation are assumptions in the Lean proof, not verified code. 07:52 — 220 of 220, and what that's worth The preregistered results — 220/220 against GPT-6.1 Sol's 217, the base model's 23, and a separate 200-problem browser test across division, roots, and GCDs. 09:28 — Perfect arithmetic, worse word problems The uncertified dialogue adapter picks the wrong computation on ~7% of held-out word problems and loses ground on fresh GSM8K items — and the hosts land their three takeaways on what the certificate does and doesn't buy. Recommended Reading Show Your Work: Scratchpads for Intermediate Computation with Language Models — The canonical 'write intermediate steps to a scratchpad' paper for multi-digit arithmetic — useful contrast with Sketchpad, which keeps the worksheet but denies the model any access to its own prior tokens. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks — The foundational work on formally verifying neural networks rather than just measuring accuracy, which is the ambition Sketchpad pursues by shrinking the input space until exhaustive checking becomes feasible. Toolformer: Language Models Can Teach Themselves to Use Tools — The mainstream alternative Eric reaches for in the opening — hand arithmetic to a calculator API — making explicit what Sketchpad gives up and gains by certifying the network instead. Exploring Length Generalization in Large Language Models — Empirical evidence that transformers trained on short problems fail on longer ones, which is precisely the failure mode Sketchpad sidesteps by making the thousandth column a repeat visit to an already-checked case.