Skip to content
Sign in
Artwork for Turing Post
Turing Post · September 9 · 33 min

Does AI Understand the Machine It Runs On? | Inside OpenAI

What if a model finds an optimization that surprises the engineers who have spent years working on that system? What, exactly, has it understood? I brought that question to OpenAI’s Phil Tillet and Matt Ferrari, whose work involves making AI cheaper and more accessible. They’re increasingly doing that work with the models themselves. Matt talks about research ideas his team used to dismiss because the engineering would be too complicated. Now they can give a model years of earlier research and ask it to explore what might work. They’re using models to help improve the smaller models behind speculative decoding, including the training process itself. So I ask whether the model is also helping choose the ideas, and how much that expands what they’re willing to try. I also ask something familiar to anyone who uses these systems: why does the same model sometimes feel different? Phil explains that even changing the order of floating-point calculations can introduce differences in its behavior. That puts a very concrete problem behind our conversation about understanding: an optimization can make the system faster and still change something you wanted to preserve. We get into how they catch those changes, and what happens when the failure is something nobody thought to test for. *We talk about:* When models became useful for engineering decisions. Why OpenAI’s models needed more control than Triton gave them. What happens inside the system after you send a request. How model-assisted kernel improvements helped cut Sol’s serving costs by 20%. Letting models investigate bugs independently, and deciding when to step in. Why successfully optimizing something can still be a waste of effort. Whether models need an internal representation of how a computer system behaves. How AI assistance opens up experiments that engineers previously couldn’t justify attempting. *Chapters:* *Follow on*: https://www.turingpost.com/ *Did you like the episode? You know the drill:* 📌 Subscribe here and here (https://www.turingpost.com/subscribe) for more conversations with the builders shaping real-world AI. 💬 Leave a comment 👍 Like it 🫶 Thank you for watching and sharing! *Guests:* Philippe (Phil) Tillet created Triton, a programming language that makes efficient GPU programming more accessible. He joined OpenAI as an intern in 2019, before it had an API or a product, and spent years improving training efficiency. His interests extend from compilers and kernels to Bertrand Russell and philosophy of mind. Matthew (Matt) Ferrari works on inference efficiency at OpenAI, across request routing, load balancing, debugging and speculative decoding. His fascination with optimization began in school, when GPU programming changed his understanding of how fast an algorithm could run. Today, he brings that curiosity to the entire system serving a model. #openai #inference #optimization

0:00-33:16
Play in

transcript

No transcript — this publisher did not publish one.

show notes

What if a model finds an optimization that surprises the engineers who have spent years working on that system? What, exactly, has it understood?


I brought that question to OpenAI’s Phil Tillet and Matt Ferrari, whose work involves making AI cheaper and more accessible. They’re increasingly doing that work with the models themselves.


Matt talks about research ideas his team used to dismiss because the engineering would be too complicated. Now they can give a model years of earlier research and ask it to explore what might work. They’re using models to help improve the smaller models behind speculative decoding, including the training process itself. So I ask whether the model is also helping choose the ideas, and how much that expands what they’re willing to try.


I also ask something familiar to anyone who uses these systems: why does the same model sometimes feel different? Phil explains that even changing the order of floating-point calculations can introduce differences in its behavior. That puts a very concrete problem behind our conversation about understanding: an optimization can make the system faster and still change something you wanted to preserve. We get into how they catch those changes, and what happens when the failure is something nobody thought to test for.


*We talk about:*

When models became useful for engineering decisions.

Why OpenAI’s models needed more control than Triton gave them.

What happens inside the system after you send a request.

How model-assisted kernel improvements helped cut Sol’s serving costs by 20%.

Letting models investigate bugs independently, and deciding when to step in.

Why successfully optimizing something can still be a waste of effort.

Whether models need an internal representation of how a computer system behaves.

How AI assistance opens up experiments that engineers previously couldn’t justify attempting.


*Chapters:*

 

*Follow on*: https://www.turingpost.com/


*Did you like the episode? You know the drill:* 

📌 Subscribe here and here (https://www.turingpost.com/subscribe) for more conversations with the builders shaping real-world AI. 

💬 Leave a comment

👍 Like it 

🫶 Thank you for watching and sharing!


*Guests:* 

Philippe (Phil) Tillet created Triton, a programming language that makes efficient GPU programming more accessible. He joined OpenAI as an intern in 2019, before it had an API or a product, and spent years improving training efficiency. His interests extend from compilers and kernels to Bertrand Russell and philosophy of mind.

Matthew (Matt) Ferrari works on inference efficiency at OpenAI, across request routing, load balancing, debugging and speculative decoding. His fascination with optimization began in school, when GPU programming changed his understanding of how fast an algorithm could run. Today, he brings that curiosity to the entire system serving a model.


#openai #inference #optimization

more episodes

All episodes