The AutoML Podcast

MLGym: A New Framework and Benchmark for Advancing AI Research Agents

AutoML is dead an LLMs have killed it? MLGym is a benchmark and framework testing this theory. Roberta Raileanu and Deepak Nathani discuss how well current LLMs are doing at solving ML tasks, what the biggest roadblocks are, and what that means for AutoML generally. Check out the paper: https://arxiv.org/pdf/2502.14499 More on Roberta: https://rraileanu.github.io/ More on Deepak: https://dnathani.net/

Leverage Foundational Models for Black-Box Optimization

Where and how can we use foundation models in AutoML? Richard Song, researcher at Google DeepMind, has some answers. Starting off from his position paper on leveraging foundation models for optimization, we chat about what makes foundation models valuable for AutoML, how the next steps could look like, but also why the community is not currently embracing the topic as much as it could. Paper Link: https://arxiv.org/abs/2405.03547 Richard's website: https://xingyousong.github.io/

Nyckel - Building an AutoML Startup

Oscar Beijbom is talking about what it's like to run an AutoML startup: Nyckel. Beyond that, we chat about the differences between academia and industry, what truly matters in application and more. Check out Nyckel at: https://www.nyckel.com/

• Transcript

Neural Architecture Search: Insights from 1000 Papers

Colin White, head of research at Abacus AI, takes us on a tour of Neural Architecture Search: its origins, important paradigms and the future of NAS in the age of LLMs. If you're looking for a broad overview of NAS, this is the podcast for you!

• Transcript

Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How

There are so many great foundation models in many different domains - but how do you choose one for your specific problem? And how can you best finetune it? Sebastian Pineda has an answer: Quicktune can help select the best model and tune it for specific use cases. Listen to find out when this will be a Huggingface feature and if hyperparameter optimization is even important in finetuning models (spoiler: very much so)!

Discovering Temporally-Aware Reinforcement Learning Algorithms

Designing algorithms by hand is hard, so Chris Lu and Matthew Jackson talk about how to meta-learn them for reinforcement learning. Many of the concepts in this episode are interesting to meta-learning approaches as a whole, though: "how expressive can we be and still perform well?", "how can we get the necessary data to generalize?" and "how do we make the resulting algorithm easy to apply in practice?" are problems that come up for any learning-based approach to AutoML and some of the topics we dive into.

X Hacking: The Threat of Misguided AutoML

AutoML can be a tool for good, but there are pitfalls along the way. Rahul Sharma and David Selby tell us about how AutoML systems can be used to give us false impressions about explainability metrics of ML systems - maliciously, but also on accident. While this episode isn't talking about a new exciting AutoML method, it can tell us a lot about what can go wrong in applying AutoML and what we should think about when we build tools for ML novices to use.

Introduction To New Co-Host, Theresa Eimer

In today's episode, we're introducing the very special Theresa Eimer to the show. Theresa will be taking over the hosting of many of the future episodes. Theresa has already recorded multiple episodes and we are stoked to air those shortly. We also spend a few moments explaining my relative absence in the last few months (since the war in the middle east erupted) and what I'm up to now. Theresa, we are all so excited to be doing this together! To learn more about Theresa, Follow her on Twitter here: https://twitter.com/The_Eimer Connect with her on LinkedIn here: https://www.linkedin.com/in/theresa-eimer-a724b5b0/ As you'll hear in the episode, she's also one of the co-organizers of COSEAL, which you can learn more about here: https://www.coseal.net/

AutoGluon: The Story

Today we're talking with Nick Erickson from AutoGluon. We discuss AutoGluon's fascinating origin story, its unique point of view, the science and engineering behind some of its unique contributions, Amazon's Machine Learning University, AutoGluon's multi-layer stack ensembler in all its detail, their feature preprocessing pipeline, their feature type inference, their adaptive approach to early stopping, controlling for inference speeds, the different multi-modal architectures, the ML culture at Amazon, the unique challenges of time series, the role of competitions, the decision to reject hyperparameter optimization, benchmarking in AutoML, what the research community can do to help industry along, AutoGluon's relationship with pre-trained tabular models like Tab-PFN, whether the rise of LLMs is likely to affect AutoGluon, what's stopping more people from adopting AutoML solutions, AutoGluon Cloud, the dream and reality of an auto-benchmarking tool, how to contribute to their project, and many, many other topics. This was one of my favorite episodes. Nick, thank you for joining! You can follow Nick on Twitter here: @innixma. And you can follow AutoGluon on GitHub here: https://github.com/autogluon. Some more resources on AutoGluon: The original AutoGluon Paper: "AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data": https://arxiv.org/abs/2003.06505 AutoML Fall School 2022 AutoGluon presentation, a good way to understand the philosophy behind AutoGluon: https://www.youtube.com/watch?v=VAAITEds-28 AutoGluon multi-modal paper: https://dl.acm.org/doi/abs/10.1145/3534678.3542616

How to Integrate Logic and Argumentation into Human-Centric AutoML

Today we're talking with Joseph Giovanelli about his work on integrating logic and argumentation into AutoML systems. Joseph is a PhD student at the University of Bologna. He was more recently in Hannover working on ethics and fairness with Marius’ team. The paper he published presents his framework, HAMLET, which stands for Human-centric AutoML via Logic and Argumentation. It allows a user to iteratively specify constraints in a formal manner and, once defined, those constraints become logical premises. Those premises, when combined together, can produce conflicts with one another, thereby reducing the search space and providing deeper intuition back to the user. To learn more about HAMLET, see the paper here: https://ceur-ws.org/Vol-3135/dataplat_short2.pdf and the repo here: https://github.com/QueueInc/HAMLET To follow Joseph on LinkedIn, see his profile here: https://www.linkedin.com/in/joseph-giovanelli/

How to Design an AutoML System using Error Decomposition

Today we're talking with Caitlin Owen, a post-doc at the University of Otago about her work on error decomposition. She recently published a paper titled "Towards Explainable AutoML Using Error Decomposition" about how a more granular view of the components of error can lead the construction of better AutoML systems. Read her paper here: https://link.springer.com/chapter/10.1007/978-3-031-22695-3_13 Follow her on Twitter here: @CaitAshfordOwen Connect with her on LinkedIn here: https://www.linkedin.com/in/caitlin-owen-5b9b08193/

The Semantic Layer and AutoML

Today we're talking with Gaurav Rao, the EVP & GM of Machine Learning and AI at AtScale, a company centered around the semantic layer. For some time now, I've been feeling that there is a deep connection between a formal articulation of business context and the realization of the dream of AutoML, so I searched for people in the space who can help shine light on this direction. Gaurav is one of the few who can speak about this. As you'll hear, he's extremely pedagogic and he's walking us through the origins of the concept, how it addresses some of the challenges that businesses face when trying to operationalize their ML, what it takes to build a universal semantic layer, how downstream ML applications are affected by the presence or absence of a semantic layer, and how the space of AutoML factors into this. Connect his Gaurav and learn more about the semantic layer through his LinkedIn: https://www.linkedin.com/in/gauravraotechenthusiast/

Foundation Models: The term and its origins

Today Ankush Garg is speaking with Rishi Bommasani, PhD student at Stanford and one of the originator of the term Foundation Models. They’re talking about the origins of the term Foundation Model, which he and his group advanced, in the paper "On the Opportunities and Risks of Foundation Models". They’ll talk about self-supervision, issues of scale, the motivation behind the terminology, the origins of the Research for Foundation Models Institute at Stanford, outcome homogenization, emergence and phase transitions, and some of the social consequences to look out for. Thank you both for this conversation. As the world is coming to terms with GPT-4, this will be increasingly relevant. Paper: On the Opportunities and Risks of Foundation Models. Rishi's twitter: @RishiBommasani Center for Research on Foundation Models (CRFM)

The Business and Engineering of AutoML Products with Raymond Peck

Today we're talking with Raymond Peck, a senior engineer and director in the AutoML space. He spent time at H2O, dotData, Alteryx and many other places. This is a fascinating conversation about the business, engineering, and science of machine learning automation in production. Learning about his experience is crucial for understanding the biography of the space. We discuss the early motivations behind AutoML, the initial value propositions that propelled the first movers in the market, the market dynamics that operated in the early days, the evolution of the relevant engineering and science, how customers evaluate AutoML tools, the role of feature engineering and relational tables, the crucial role that explainability plays in AutoML, and many more topics. Raymond is a prolific writer on LinkedIn. You should follow him here: https://www.linkedin.com/in/raymondpeck/.

TabPFN: A Revolution in AutoML?

Today we’re talking to Noah Hollmann and Samuel Muller about their paper on TabPFN - which is an incredible spin on AutoML based on Bayesian inference and transformers. [Quick note on audio quality]: Some of the tracks have not recorded perfectly but I felt that the content there was too important not to release. Sorry for any ear-strain! In the episode, we spend some time discussing posterior predictive probabilities before discussing how exactly they’ve pre-fitted their network, how they got their training data, what the network looks like, and how the system is performing. To give you a taste of it, on datasets up to 1,000 training instances and 100 features, it takes less than a second to train and predict a classifier! Read their paper here: https://arxiv.org/pdf/2207.01848.pdf Follow Samuel on Twitter, here: https://twitter.com/SamuelMullr Follow Noah on Twitter, here: https://twitter.com/noahholl

How financial institutions manage model risk

Today we’re talking to Sean Sexton, the Director of Modeling and Analytics Consulting at KPMG, about the role of models in financial institutions and how the risks associated with them is managed. This turned out to be an incredibly deep and interesting topic, and we really only scratched the surface of it. Sean has a unique ability to summarize developments in an entire space. If you're interested to learn more about modeling in financial institutions and about the history of how we got here, you should definitely study his dissertation, on managing model risk, here: https://macsphere.mcmaster.ca/handle/11375/28049

How to solve dynamical systems by fusing data and mechanism

Today we’re talking to Matt Levine. Matt is a PhD student in computing and mathematical sciences at Caltech, and he focuses on improving the prediction and inference of physical systems by blending together both mechanistic modeling and machine learning. This episode is one of my favorites: we go pretty deep into dynamical systems, and into Matt's new framework for solving them by blending traditional, mechanistic, approaches with machine learning. This is a fascinating use of machine learning, and hopefully gets us one step closer to the automation of science, in general. A Framework for Machine Learning of Model Error in Dynamical Systems - https://arxiv.org/abs/2107.06658 Related works Autodifferentiable Ensemble Kalman Filters - https://epubs.siam.org/doi/abs/10.1137/21M1434477 Universal Differential Equations for Scientific Machine Learning - https://arxiv.org/abs/2001.04385 Continuous-time nonlinear signal processing: a neural network based approach for gray box identification - https://ieeexplore.ieee.org/document/366006 A generalised approach to process state estimation using hybrid artificial neural network/mechanistic models - https://www.sciencedirect.com/science/article/abs/pii/S0098135496003365

DASH: How to Search Over Convolutions

Today we’re chatting with Junhong Shen, a PhD student at Carnegie Mellon. Junhong and her team are working on the generalizability of NAS algorithms across a diverse set of tasks. Today we'll be talking about DASH, a NAS algorithm that takes diversity of tasks at its center. In order to implement DASH, Junhong and her team implemented three clever ideas that she'll share with us. Efficient Architecture Search for Diverse Tasks - https://arxiv.org/pdf/2204.07554.pdf Tackling Diverse Tasks with Neural Architecture Search - https://blog.ml.cmu.edu/2022/10/14/tackling-diverse-tasks-with-neural-architecture-search/ Does AutoML work for diverse tasks? - https://blog.ml.cmu.edu/2022/07/07/automl-for-diverse-tasks Follow Junhong @JunhongShen1

Human-Centered AutoML: The New Paradigm

Today we're speaking with Marius Lindauer and it is certainly one of my favorite episodes! As you’ll hear, Marius is full of ideas for where AutoML systems can and should go. These ideas are crystallized in a blog-post, published here: https://www.automl.org/rethinking-automl-advancing-from-a-machine-centered-to-human-centered-paradigm/ If you’re searching for research directions, this conversation left me with dozens of ideas. Marius and his team are doing phenomenal work to make AutoML systems more trustworthy and more human-centric. We will be reviewing content from the following papers: Bayesian Optimization with a Prior for the Optimum - https://arxiv.org/abs/2006.14608 Explaining Hyperparameter Optimization via Partial Dependence Plots - https://arxiv.org/abs/2111.04820 Enhancing Explainability of Hyperparameter Optimization via Bayesian Algorithm Execution - https://arxiv.org/abs/2206.05447 πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization - https://arxiv.org/abs/2204.11051 Follow Marius on Twitter here - https://twitter.com/LindauerMarius, and AutoML.org on Twitter here - https://twitter.com/AutoML_org. To help Marius complete the survey he mentioned, please visit the link here - https://www.soscisurvey.de/hpo-method-validation/. To learn more about AutoML, visit AutoML.org, here - https://www.automl.org/

BERT-Sort: How to use language models to semantically order categorical values

Today Ankush Garg is talking to Mehdi Bahrami about his recent project: BERT-Sort. BERT-Sort is an example of how large language models can add useful context to tabular datasets, and to AutoML systems. Mehdi is a Member of Research Staff at Fujitsu and, as he describes, he began using AutoML systems for his research, yet he came across some crucial limitations of existing solutions. The modifications he made highlight a promising future for the relationship between language models and AutoML. This is a direction we're going to continue to explore on the show. References: BERT-Sort: A Zero-shot MLM Semantic Encoder on Ordinal Features for AutoML - https://proceedings.mlr.press/v188/bahrami22a.html PyTorrent: A Python Library Corpus for Large-scale Language Models: https://arxiv.org/abs/2110.01710 AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval Models: https://arxiv.org/abs/2110.08512