Machine Learning Street Talk (MLST)

Welcome! We engage in fascinating discussions with pre-eminent figures in the AI field. Our flagship show covers current affairs in AI, cognitive science, neuroscience and philosophy of mind with in-depth analysis. Our approach is unrivalled in terms of scope and rigour – we believe in intellectual diversity in AI, and we touch on all of the main ideas in the field with the hype surgically removed. MLST is run by Tim Scarfe, Ph.D (https://www.linkedin.com/in/ecsquizor/) and features regular appearances from MIT Doctor of Philosophy Keith Duggar (https://www.linkedin.com/in/dr-keith-duggar/).

Subscribe Listen in podnod RSS feed Show website

257 episodes
Updated Friday

Episodes257

Newest first

Jul 25, 2021 · 2 hr 31 min
#57 - Prof. Melanie Mitchell - Why AI is harder than we think
Since its beginning in the 1950s, the field of artificial intelligence has vacillated between periods of optimistic predictions and massive investment and periods of disappointment, loss of confidence, and reduced funding. Even with today’s seemingly fast pace of AI breakthroughs, the development of long-promised technologies such as self-driving cars, housekeeping robots, and conversational companions has turned out to be much harder than many people expected. Professor Melanie Mitchell thinks one reason for these repeating cycles is our limited understanding of the nature and complexity of intelligence itself. YT vid- https://www.youtube.com/watch?v=A8m1Oqz2HKc Main show kick off [00:26:51] Panel: Dr. Tim Scarfe, Dr. Keith Duggar, Letitia Parcalabescu (https://www.youtube.com/c/AICoffeeBreak/)
Jul 8, 2021 · 1 hr 11 min
#56 - Dr. Walid Saba, Gadi Singer, Prof. J. Mark Bishop (Panel discussion)
It has been over three decades since the statistical revolution overtook AI by a storm and over two decades since deep learning (DL) helped usher the latest resurgence of artificial intelligence (AI). However, the disappointing progress in conversational agents, NLU, and self-driving cars, has made it clear that progress has not lived up to the promise of these empirical and data-driven methods. DARPA has suggested that it is time for a third wave in AI, one that would be characterized by hybrid models – models that combine knowledge-based approaches with data-driven machine learning techniques. Joining us on this panel discussion is polymath and linguist Walid Saba - Co-founder ONTOLOGIK.AI, Gadi Singer - VP & Director, Cognitive Computing Research, Intel Labs and J. Mark Bishop - Professor of Cognitive Computing (Emeritus), Goldsmiths, University of London and Scientific Adviser to FACT360. Moderated by Dr. Keith Duggar and Dr. Tim Scarfe https://www.linkedin.com/in/gadi-singer/ https://www.linkedin.com/in/walidsaba/ https://www.linkedin.com/in/profjmarkbishop/ #machinelearning #artificialintelligence
Jun 21, 2021 · 1 hr 36 min
#55 Self-Supervised Vision Models (Dr. Ishan Misra - FAIR).
Dr. Ishan Misra is a Research Scientist at Facebook AI Research where he works on Computer Vision and Machine Learning. His main research interest is reducing the need for human supervision, and indeed, human knowledge in visual learning systems. He finished his PhD at the Robotics Institute at Carnegie Mellon. He has done stints at Microsoft Research, INRIA and Yale. His bachelors is in computer science where he achieved the highest GPA in his cohort. Ishan is fast becoming a prolific scientist, already with more than 3000 citations under his belt and co-authoring with Yann LeCun; the godfather of deep learning. Today though we will be focusing an exciting cluster of recent papers around unsupervised representation learning for computer vision released from FAIR. These are; DINO: Emerging Properties in Self-Supervised Vision Transformers, BARLOW TWINS: Self-Supervised Learning via Redundancy Reduction and PAWS: Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples. All of these papers are hot off the press, just being officially released in the last month or so. Many of you will remember PIRL: Self-Supervised Learning of Pretext-Invariant Representations which Ishan was the primary author of in 2019. References; Shuffle and Learn - https://arxiv.org/abs/1603.08561 DepthContrast - https://arxiv.org/abs/2101.02691 DINO - https://arxiv.org/abs/2104.14294 Barlow Twins - https://arxiv.org/abs/2103.03230 SwAV - https://arxiv.org/abs/2006.09882 PIRL - https://arxiv.org/abs/1912.01991 AVID - https://arxiv.org/abs/2004.12943 (best paper candidate at CVPR'21 (just announced over the weekend) - http://cvpr2021.thecvf.com/node/290) Alexei (Alyosha) Efros http://people.eecs.berkeley.edu/~efros/ http://www.cs.cmu.edu/~tmalisie/projects/nips09/ Exemplar networks https://arxiv.org/abs/1406.6909 The bitter lesson - Rich Sutton http://www.incompleteideas.net/IncIdeas/BitterLesson.html Machine Teaching: A New Paradigm for Building Machine Learning Systems https://arxiv.org/abs/1707.06742 POET https://arxiv.org/pdf/1901.01753.pdf
Jun 4, 2021 · 2 hr 24 min
#54 Gary Marcus and Luis Lamb - Neurosymbolic models
Professor Gary Marcus is a scientist, best-selling author, and entrepreneur. He is Founder and CEO of Robust.AI, and was Founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016. Gary said in his recent next decade paper that — without us, or other creatures like us, the world would continue to exist, but it would not be described, distilled, or understood. Human lives are filled with abstraction and causal description. This is so powerful. Francois Chollet the other week said that intelligence is literally sensitivity to abstract analogies, and that is all there is to it. It's almost as if one of the most important features of intelligence is to be able to abstract knowledge, this drives the generalisation which will allow you to mine previous experience to make sense of many future novel situations. Also joining us today is Professor Luis Lamb — Secretary of Innovation for Science and Technology of the State of Rio Grande do Sul, Brazil. His Research Interests are Machine Learning and Reasoning, Neuro-Symbolic Computing, Logic in Computation and Artificial Intelligence, Cognitive and Neural Computation and also AI Ethics and Social Computing. Luis released his new paper Neurosymbolic AI: the third wave at the end of last year. It beautifully articulated the key ingredients needed in the next generation of AI systems, integrating type 1 and type 2 approaches to AI and it summarises all the of the achievements of the last 20 years of research. We cover a lot of ground in today's show. Explaining the limitations of deep learning, Rich Sutton's the bitter lesson and "reward is enough", and the semantic foundation which is required for us to build robust AI.
May 19, 2021 · 2 hr 17 min
#53 Quantum Natural Language Processing - Prof. Bob Coecke (Oxford)
Bob Coercke is a celebrated physicist, he's been a Physics and Quantum professor at Oxford University for the last 20 years. He is particularly interested in Structure which is to say, Logic, Order, and Category Theory. He is well known for work involving compositional distributional models of natural language meaning and he is also fascinated with understanding how our brains work. Bob was recently appointed as the Chief Scientist at Cambridge Quantum Computing. Bob thinks that interactions between systems in Quantum Mechanics carries naturally over to how word meanings interact in natural language. Bob argues that this interaction embodies the phenomenon of quantum teleportation. Bob invented ZX-calculus, a graphical calculus for revealing the compositional structure inside quantum circuits - to show entanglement states and protocols in a visually succinct but logically complete way. Von Neumann himself didn't even like his own original symbolic formalism of quantum theory, despite it being widely used! We hope you enjoy this fascinating conversation which might give you a lot of insight into natural language processing. Tim Intro [00:00:00] The topological brain (Post-record button skit) [00:13:22] Show kick off [00:19:31] Bob introduction [00:22:37] Changing culture in universities [00:24:51] Machine Learning is like electricity [00:31:50] NLP -- what is Bob's Quantum conception? [00:34:50] The missing text problem [00:52:59] Can statistical induction be trusted? [00:59:49] On pragmatism and hybrid systems [01:04:42] Parlour tricks, parsing and information flows [01:07:43] How much human input is required with Bob's method? [01:11:29] Reality, meaning, structure and language [01:14:42] Replacing complexity with quantum entanglement, emergent complexity [01:17:45] Loading quantum data requires machine learning [01:19:49] QC is happy math coincidence for NLP [01:22:30] The Theory of English (ToE) [01:28:23] ... or can we learn the ToE? [01:29:56] How did diagrammatic quantum calculus come about? [01:31:04 The state of quantum computing today [01:37:49] NLP on QC might be doable even in the NISQ era [01:40:48] Hype and private investment are driving progress [01:48:34] Crypto discussion (moved to post-show) [01:50:38] Kilcher is in a startup (moved to post show) [01:53:40 Debrief [01:55:26]
May 1, 2021 · 1 hr 48 min
#52 - Unadversarial Examples (Hadi Salman, MIT)
Performing reliably on unseen or shifting data distributions is a difficult challenge for modern vision systems, even slight corruptions or transformations of images are enough to slash the accuracy of state-of-the-art classifiers. When an adversary is allowed to modify an input image directly, models can be manipulated into predicting anything even when there is no perceptible change, this is known an adversarial example. The ideal definition of an adversarial example is when humans consistently say two pictures are the same but a machine disagrees. Hadi Salman, a Ph.D student at MIT (ex-Uber and Microsoft Research) started thinking about how adversarial robustness could be leveraged beyond security. He realised that the phenomenon of adversarial examples could actually be turned upside down to lead to more robust models instead of breaking them. Hadi actually utilized the brittleness of neural networks to design unadversarial examples or robust objects which_ are objects designed specifically to be robustly recognized by neural networks. Introduction [00:00:00] DR KILCHER'S PHD HAT [00:11:18] Main Introduction [00:11:38] Hadi's Introduction [00:14:43] More robust models == transfer better [00:46:41] Features not bugs paper [00:49:13] Manifolds [00:55:51] Robustness and Transferability [00:58:00] Do non-robust features generalize worse than robust? [00:59:52] The unreasonable predicament of entangled features [01:01:57] We can only find adversarial examples in the vicinity [01:09:30] Certifiability of models for robustness [01:13:55] Carlini is coming for you! And we are screwed [01:23:21] Distribution shift and corruptions are a bigger problem than adversarial examples [01:25:34] All roads lead to generalization [01:26:47] Unadversarial examples [01:27:26]
Apr 16, 2021 · 2 hr 1 min
#51 Francois Chollet - Intelligence and Generalisation
In today's show we are joined by Francois Chollet, I have been inspired by Francois ever since I read his Deep Learning with Python book and started using the Keras library which he invented many, many years ago. Francois has a clarity of thought that I've never seen in any other human being! He has extremely interesting views on intelligence as generalisation, abstraction and an information conversation ratio. He wrote on the measure of intelligence at the end of 2019 and it had a huge impact on my thinking. He thinks that NNs can only model continuous problems, which have a smooth learnable manifold and that many "type 2" problems which involve reasoning and/or planning are not suitable for NNs. He thinks that many problems have type 1 and type 2 enmeshed together. He thinks that the future of AI must include program synthesis to allow us to generalise broadly from a few examples, but the search could be guided by neural networks because the search space is interpolative to some extent. https://youtu.be/J0p_thJJnoo Tim's Whimsical notes; https://whimsical.com/chollet-show-QQ2atZUoRR3yFDsxKVzCbj
Apr 4, 2021 · 1 hr 33 min
#50 Christian Szegedy - Formal Reasoning, Program Synthesis
Dr. Christian Szegedy from Google Research is a deep learning heavyweight. He invented adversarial examples, one of the first object detection algorithms, the inceptionnet architecture, and co-invented batchnorm. He thinks that if you bet on computers and software in 1990 you would have been as right as if you bet on AI now. But he thinks that we have been programming computers the same way since the 1950s and there has been a huge stagnation ever since. Mathematics is the process of taking a fuzzy thought and formalising it. But could we automate that? Could we create a system which will act like a super human mathematician but you can talk to it in natural language? This is what Christian calls autoformalisation. Christian thinks that automating many of the things we do in mathematics is the first step towards software synthesis and building human-level AGI. Mathematics ability is the litmus test for general reasoning ability. Christian has a fascinating take on transformers too. With Yannic Lightspeed Kilcher and Dr. Mathew Salvaris Whimsical Canvas with Tim's Notes: https://whimsical.com/mar-26th-christian-szegedy-CpgGhnEYDBrDMFoATU6XYC YouTube version (with detailed table of contents) https://youtu.be/ehNGGYFO6ms
Mar 23, 2021 · 1 hr 25 min
#49 - Meta-Gradients in RL - Dr. Tom Zahavy (DeepMind)
The race is on, we are on a collective mission to understand and create artificial general intelligence. Dr. Tom Zahavy, a Research Scientist at DeepMind thinks that reinforcement learning is the most general learning framework that we have today, and in his opinion it could lead to artificial general intelligence. He thinks there are no tasks which could not be solved by simply maximising a reward. Back in 2012 when Tom was an undergraduate, before the deep learning revolution he attended an online lecture on how CNNs automatically discover representations. This was an epiphany for Tom. He decided in that very moment that he was going to become an ML researcher. Tom's view is that the ability to recognise patterns and discover structure is the most important aspect of intelligence. This has been his quest ever since. He is particularly focused on using diversity preservation and metagradients to discover this structure. In this discussion we dive deep into meta gradients in reinforcement learning. Video version and TOC @ https://www.youtube.com/watch?v=hfaZwgk_iS0
Mar 16, 2021 · 37 min
#48 Machine Learning Security - Andy Smith
First episode in a series we are doing on ML DevOps. Starting with the thing which nobody seems to be talking about enough, security! We chat with cyber security expert Andy Smith about threat modelling and trust boundaries for an ML DevOps system. Intro [00:00:00] ML DevOps - a security perspective [00:00:50] Threat Modelling [00:03:03] Adversarial examples? [00:11:27] Nobody understands the whole stack [00:13:53] On the size of the state space, the element of unpredictability [00:18:32] Threat modelling in more detail [00:21:17] Trust boundaries for an ML DevOps system [00:25:45] Andy has a YouTube channel on cyber security! Check it out @ https://www.youtube.com/channel/UCywP24ly6h6NTusX88TQKTQ https://www.linkedin.com/in/andysmith-uk/ Video version: https://youtu.be/7Tz-3S4lypI
Mar 14, 2021 · 1 hr 40 min
047 Interpretable Machine Learning - Christoph Molnar
Christoph Molnar is one of the main people to know in the space of interpretable ML. In 2018 he released the first version of his incredible online book, interpretable machine learning. Interpretability is often a deciding factor when a machine learning (ML) model is used in a product, a decision process, or in research. Interpretability methods can be used to discover knowledge, to debug or justify the model and its predictions, and to control and improve the model, reason about potential bias in models as well as increase the social acceptance of models. But Interpretability methods can also be quite esoteric, add an additional layer of complexity and potential pitfalls and requires expert knowledge to understand. Is it even possible to understand complex models or even humans for that matter in any meaningful way? Introduction to IML [00:00:00] Show Kickoff [00:13:28] What makes a good explanation? [00:15:51] Quantification of how good an explanation is [00:19:59] Knowledge of the pitfalls of IML [00:22:14] Are linear models even interpretable? [00:24:26] Complex Math models to explain Complex Math models? [00:27:04] Saliency maps are glorified edge detectors [00:28:35] Challenge on IML -- feature dependence [00:36:46] Don't leap to using a complex model! Surrogate models can be too dumb [00:40:52] On airplane pilots. Seeking to understand vs testing [00:44:09] IML Could help us make better models or lead a better life [00:51:53] Lack of statistical rigor and quantification of uncertainty [00:55:35] On Causality [01:01:09] Broadening out the discussion to the process or institutional level [01:08:53] No focus on fairness / ethics? [01:11:44] Is it possible to condition ML model training on IML metrics ? [01:15:27] Where is IML going? Some of the esoterica of the IML methods [01:18:35] You can't compress information without common knowledge, the latter becomes the bottleneck [01:23:25] IML methods used non-interactively? Making IML an engineering discipline [01:31:10] Tim Postscript -- on the lack of effective corporate operating models for IML, security, engineering and ethics [01:36:34] Explanation in Artificial Intelligence: Insights from the Social Sciences (Tim Miller 2018) https://arxiv.org/pdf/1706.07269.pdf Seven Myths in Machine Learning Research (Chang 19) Myth 7: Saliency maps are robust ways to interpret neural networks https://arxiv.org/pdf/1902.06789.pdf Sanity Checks for Saliency Maps (Adebayo 2020) https://arxiv.org/pdf/1810.03292.pdf Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. https://christophm.github.io/interpretable-ml-book/ Christoph Molnar: https://www.linkedin.com/in/christoph-molnar-63777189/ https://machine-master.blogspot.com/ https://twitter.com/ChristophMolnar Please show your appreciation and buy Christoph's book here; https://www.lulu.com/shop/christoph-molnar/interpretable-machine-learning/paperback/product-24449081.html?page=1&pageSize=4 Panel: Connor Tann https://www.linkedin.com/in/connor-tann-a92906a1/ Dr. Tim Scarfe Dr. Keith Duggar Video version: https://youtu.be/0LIACHcxpHU
Mar 6, 2021 · 1 hr 39 min
#046 The Great ML Stagnation (Mark Saroufim and Dr. Mathew Salvaris)
Academics think of themselves as trailblazers, explorers — seekers of the truth. Any fundamental discovery involves a significant degree of risk. If an idea is guaranteed to work then it moves from the realm of research to engineering. Unfortunately, this also means that most research careers will invariably be failures at least if failures are measured via “objective” metrics like citations. Today we discuss the recent article from Mark Saroufim called Machine Learning: the great stagnation. We discuss the rise of gentleman scientists, fake rigor, incentives in ML, SOTA-chasing, "graduate student descent", distribution of talent in ML and how to learn effectively. With special guest interviewer Mat Salvaris. Machine learning: the great stagnation [00:00:00] Main show kick off [00:16:30] Great stagnation article / Bad incentive systems in academia [00:18:24] OpenAI is a media business [00:19:48] Incentive structures in academia [00:22:13] SOTA chasing [00:24:47] F You Money [00:28:53] Research grants and gentlemen scientists [00:29:13] Following your own gradient of interest and making a contribution [00:33:27] Marketing yourself to be successful [00:37:07] Tech companies create the bad incentives [00:42:20] GPT3 was sota chasing but it seemed really... "good"? Scaling laws? [00:51:09] Dota / game AI [00:58:39] Hard to go it alone? [01:02:08] Reaching out to people [01:09:21] Willingness to be wrong [01:13:14] Distribution of talent / tech interviews [01:18:30] What should you read online and how to learn? Sharing your stuff online and finding your niece [01:25:52] Mark Saroufim: https://marksaroufim.substack.com/ http://robotoverlordmanual.com/ https://twitter.com/marksaroufim https://www.youtube.com/marksaroufim Dr. Mathew Salvaris: https://www.linkedin.com/in/drmathewsalvaris/ https://twitter.com/MSalvaris
Feb 28, 2021 · 2 hr 30 min
#045 Microsoft's Platform for Reinforcement Learning (Bonsai)
Microsoft has an interesting strategy with their new “autonomous systems” technology also known as Project Bonsai. They want to create an interface to abstract away the complexity and esoterica of deep reinforcement learning. They want to fuse together expert knowledge and artificial intelligence all on one platform, so that complex problems can be decomposed into simpler ones. They want to take machine learning Ph.Ds out of the equation and make autonomous systems engineering look more like a traditional software engineering process. It is an ambitious undertaking, but interesting. Reinforcement learning is extremely difficult (as I cover in the video), and if you don’t have a team of RL Ph.Ds with tech industry experience, you shouldn’t even consider doing it yourself. This is our take on it! There are 3 chapters in this video; Chapter 1: Tim's intro and take on RL being hard, intro to Bonsai and machine teaching Chapter 2: Interview with Scott Stanfield [recorded Jan 2020] 00:56:41 Chapter 3: Traditional street talk episode [recorded Dec 2020] 01:38:13 This is *not* an official communication from Microsoft, all personal opinions. There is no MS-confidential information in this video. With: Scott Stanfield https://twitter.com/seesharp Megan Bloemsma https://twitter.com/BloemsmaMegan Gurdeep Pall (he has not validated anything we have said in this video or been involved in the creation of it) https://www.linkedin.com/in/gurdeep-pall-0aa639bb/ Panel: Dr. Keith Duggar Dr. Tim Scarfe Yannic Kilcher
Feb 25, 2021 · 52 min
#044 - Data-efficient Image Transformers (Hugo Touvron)
Today we are going to talk about the *Data-efficient image Transformers paper or (DeiT) which Hugo is the primary author of. One of the recipes of success for vision models since the DL revolution began has been the availability of large training sets. CNNs have been optimized for almost a decade now, including through extensive architecture search which is prone to overfitting. Motivated by the success of transformers-based models in Natural Language Processing there has been increasing attention in applying these approaches to vision models. Hugo and his collaborators used a different training strategy and a new distillation token to get a massive increase in sample efficiency with image transformers. 00:00:00 Introduction 00:06:33 Data augmentation is all you need 00:09:53 Now the image patches are the convolutions though? 00:12:16 Where are those inductive biases hiding? 00:15:46 Distillation token 00:21:01 Why different resolutions on training 00:24:14 How data efficient can we get? 00:26:47 Out of domain generalisation 00:28:22 Why are transformers data efficient at all? Learning invariances 00:32:04 Is data augmentation cheating? 00:33:25 Distillation strategies - matching the intermediatae teacher representation as well as output 00:35:49 Do ML models learn the same thing for a problem? 00:39:01 How is it like at Facebook AI? 00:41:17 How long is the PhD programme? 00:42:03 Other interests outside of transformers? 00:43:18 Transformers for Vision and Language 00:47:40 Could we improve transformers models? (Hybrid models) 00:49:03 Biggest challenges in AI? 00:50:52 How far can we go with data driven approach?
Feb 19, 2021 · 1 hr 35 min
#043 Prof J. Mark Bishop - Artificial Intelligence Is Stupid and Causal Reasoning won't fix it.
Professor Mark Bishop does not think that computers can be conscious or have phenomenological states of consciousness unless we are willing to accept panpsychism which is idea that mentality is fundamental and ubiquitous in the natural world, or put simply, that your goldfish and everything else for that matter has a mind. Panpsychism postulates that distinctions between intelligences are largely arbitrary. Mark’s work in the ‘philosophy of AI’ led to an influential critique of computational approaches to Artificial Intelligence through a thorough examination of John Searle's 'Chinese Room Argument' Mark just published a paper called artificial intelligence is stupid and causal reasoning wont fix it. He makes it clear in this paper that in his opinion computers will never be able to compute everything, understand anything, or feel anything. 00:00:00 Tim Intro 00:15:04 Intro 00:18:49 Introduction to Marks ideas 00:25:49 Some problems are not computable 00:29:57 the dancing was Pixies fallacy 00:32:36 The observer relative problem, and its all in the mapping 00:43:03 Conscious Experience 00:53:30 Intelligence without representation, consciousness is something that we do 01:02:36 Consciousness helps us to act autonomously 01:05:13 The Chinese room argument 01:14:58 Simulation argument and computation doesn't have phenomenal consciousness 01:17:44 Language informs our colour perception 01:23:11 We have our own distinct ontologies 01:27:12 Kurt Gödel, Turing and Penrose and the implications of their work
Feb 11, 2021 · 1 hr 33 min
#042 - Pedro Domingos - Ethics and Cancel Culture
Today we have professor Pedro Domingos and we are going to talk about activism in machine learning, cancel culture, AI ethics and kernels. In Pedro's book the master algorithm, he segmented the AI community into 5 distinct tribes with 5 unique identities (and before you ask, no the irony of an anti-identitarian doing do was not lost on us!). Pedro recently published an article in Quillette called Beating Back Cancel Culture: A Case Study from the Field of Artificial Intelligence. Domingos has railed against political activism in the machine learning community and cancel culture. Recently Pedro was involved in a controversy where he asserted the NeurIPS broader impact statements are an ideological filter mechanism. Important Disclaimer: All views expressed are personal opinions. 00:00:00 Caveating 00:04:08 Main intro 00:07:44 Cancelling culture is a culture and intellectual weakness 00:12:26 Is cancel culture a post-modern religion? 00:24:46 Should we have gateways and gatekeepers? 00:29:30 Does everything require broader impact statements? 00:33:55 We are stifling diversity (of thought) not promoting it. 00:39:09 What is fair and how to do fair? 00:45:11 Models can introduce biases by compressing away minority data 00:48:36 Accurate but unequal soap dispensers 00:53:55 Agendas are not even self-consistent 00:56:42 Is vs Ought: all variables should be used for Is 01:00:38 Fighting back cancellation with cancellation? 01:10:01 Intent and degree matter in right vs wrong. 01:11:08 Limiting principles matter 01:15:10 Gradient descent and kernels 01:20:16 Training Journey matter more than Destination 01:24:36 Can training paths teach us about symmetry? 01:28:37 What is the most promising path to AGI? 01:31:29 Intelligence will lose its mystery
Feb 3, 2021 · 1 hr 26 min
#041 - Biologically Plausible Neural Networks - Dr. Simon Stringer
Dr. Simon Stringer. Obtained his Ph.D in mathematical state space control theory and has been a Senior Research Fellow at Oxford University for over 27 years. Simon is the director of the the Oxford Centre for Theoretical Neuroscience and Artificial Intelligence, which is based within the Oxford University Department of Experimental Psychology. His department covers vision, spatial processing, motor function, language and consciousness -- in particular -- how the primate visual system learns to make sense of complex natural scenes. Dr. Stringers laboratory houses a team of theoreticians, who are developing computer models of a range of different aspects of brain function. Simon's lab is investigating the neural and synaptic dynamics that underpin brain function. An important matter here is the The feature-binding problem which concerns how the visual system represents the hierarchical relationships between features. the visual system must represent hierarchical binding relations across the entire visual field at every spatial scale and level in the hierarchy of visual primitives. We discuss the emergence of self-organised behaviour, complex information processing, invariant sensory representations and hierarchical feature binding which emerges when you build biologically plausible neural networks with temporal spiking dynamics. 00:00:09 Tim Intro 00:09:31 Show kickoff 00:14:37 Hierarchical Feature binding and timing of action potentials 00:30:16 Hebb to Spike-timing-dependent plasticity (STDP) 00:35:27 Encoding of shape primitives 00:38:50 Is imagination working in the same place in the brain 00:41:12 Compare to supervised CNNs 00:45:59 Speech recognition, motor system, learning mazes 00:49:28 How practical are these spiking NNs 00:50:19 Why simulate the human brain 00:52:46 How much computational power do you gain from differential timings 00:55:08 Adversarial inputs 00:59:41 Generative / causal component needed? 01:01:46 Modalities of processing i.e. language 01:03:42 Understanding 01:04:37 Human hardware 01:06:19 Roadmap of NNs? 01:10:36 Intepretability methods for these new models 01:13:03 Won't GPT just scale and do this anyway? 01:15:51 What about trace learning and transformation learning 01:18:50 Categories of invariance 01:19:47 Biological plausibility https://www.youtube.com/watch?v=aisgNLypUKs
Jan 31, 2021 · 1 hr 36 min
#040 - Adversarial Examples (Dr. Nicholas Carlini, Dr. Wieland Brendel, Florian Tramèr)
Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. there's good reason to believe neural networks look at very different features than we would have expected. As articulated in the 2019 "features not bugs" paper Adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. Adversarial examples don't just affect deep learning models. A cottage industry has sprung up around Threat Modeling in AI and ML Systems and their dependencies. Joining us this evening are some of currently leading researchers in adversarial examples; Florian Tramèr - A fifth year PhD student in Computer Science at Stanford University https://floriantramer.com/ https://twitter.com/florian_tramer Dr. Wieland Brendel - Machine Learning Researcher at the University of Tübingen & Co-Founder of layer7.ai https://medium.com/@wielandbr https://twitter.com/wielandbr Dr. Nicholas Carlini - Research scientist at Google Brain working in that exciting space between machine learning and computer security. https://nicholas.carlini.com/ We really hope you enjoy the conversation, remember to subscribe! Yannic Intro [00:00:00] Tim Intro [00:04:07] Threat Taxonomy [00:09:00] Main show intro [00:11:30] Whats wrong with Neural Networks? [00:14:52] The role of memorization [00:19:51] Anthropomorphization of models [00:22:42] Whats the harm really though / focusing on actual ML security risks [00:27:03] Shortcut learning / OOD generalization [00:36:18] Human generalization [00:40:11] An existential problem in DL getting the models to learn what we want? [00:41:39] Defenses to adversarial examples [00:47:15] What if we had all the data and the labels? Still problems? [00:54:28] Defenses are easily broken [01:00:24] Self deception in academia [01:06:46] ML Security [01:28:15] https://www.youtube.com/watch?v=2PenK06tvE4
Jan 23, 2021 · 1 hr 58 min
#039 - Lena Voita - NLP
ena Voita is a Ph.D. student at the University of Edinburgh and University of Amsterdam. Previously, She was a research scientist at Yandex Research and worked closely with the Yandex Translate team. She still teaches NLP at the Yandex School of Data Analysis. She has created an exciting new NLP course on her website lena-voita.github.io which you folks need to check out! She has one of the most well presented blogs we have ever seen, where she discusses her research in an easily digestable manner. Lena has been investigating many fascinating topics in machine learning and NLP. Today we are going to talk about three of her papers and corresponding blog articles; Source and Target Contributions to NMT Predictions -- Where she talks about the influential dichotomy between the source and the prefix of neural translation models. https://arxiv.org/pdf/2010.10907.pdf https://lena-voita.github.io/posts/source_target_contributions_to_nmt.html Information-Theoretic Probing with MDL -- Where Lena proposes a technique of evaluating a model using the minimum description length or Kolmogorov complexity of labels given representations rather than something basic like accuracy https://arxiv.org/pdf/2003.12298.pdf https://lena-voita.github.io/posts/mdl_probes.html Evolution of Representations in the Transformer - Lena investigates the evolution of representations of individual tokens in Transformers -- trained with different training objectives (MT, LM, MLM) https://arxiv.org/abs/1909.01380 https://lena-voita.github.io/posts/emnlp19_evolution.html Panel Dr. Tim Scarfe, Yannic Kilcher, Sayak Paul 00:00:00 Kenneth Stanley / Greatness can not be planned house keeping 00:21:09 Kilcher intro 00:28:54 Hello Lena 00:29:21 Tim - Lenas NMT paper 00:35:26 Tim - Minimum Description Length / Probe paper 00:40:12 Tim - Evolution of representations 00:46:40 Lenas NLP course 00:49:18 The peppermint tea situation 00:49:28 Main Show Kick Off 00:50:22 Hallucination vs exposure bias 00:53:04 Lenas focus on explaining the models not SOTA chasing 00:56:34 Probes paper and NLP intepretability 01:02:18 Why standard probing doesnt work 01:12:12 Evolutions of representations paper 01:23:53 BERTScore and BERT Rediscovers the Classical NLP Pipeline paper 01:25:10 Is the shifting encoding context because of BERT bidirectionality 01:26:43 Objective defines which information we lose on input 01:27:59 How influential is the dataset? 01:29:42 Where is the community going wrong? 01:31:55 Thoughts on GOFAI/Understanding in NLP? 01:36:38 Lena's NLP course 01:47:40 How to foster better learning / understanding 01:52:17 Lena's toolset and languages 01:54:12 Mathematics is all you need 01:56:03 Programming languages https://lena-voita.github.io/ https://www.linkedin.com/in/elena-voita/ https://scholar.google.com/citations?user=EcN9o7kAAAAJ&hl=ja https://twitter.com/lena_voita
Jan 20, 2021 · 2 hr 46 min
#038 - Professor Kenneth Stanley - Why Greatness Cannot Be Planned
Professor Kenneth Stanley is currently a research science manager at OpenAI in San Fransisco. We've Been dreaming about getting Kenneth on the show since the very begininning of Machine Learning Street Talk. Some of you might recall that our first ever show was on the enhanced POET paper, of course Kenneth had his hands all over it. He's been cited over 16000 times, his most popular paper with over 3K citations was the NEAT algorithm. His interests are neuroevolution, open-endedness, NNs, artificial life, and AI. He invented the concept of novelty search with no clearly defined objective. His key idea is that there is a tyranny of objectives prevailing in every aspect of our lives, society and indeed our algorithms. Crucially, these objectives produce convergent behaviour and thinking and distract us from discovering stepping stones which will lead to greatness. He thinks that this monotonic objective obsession, this idea that we need to continue to improve benchmarks every year is dangerous. He wrote about this in detail in his recent book "greatness can not be planned" which will be the main topic of discussion in the show. We also cover his ideas on open endedness in machine learning. 00:00:00 Intro to Kenneth 00:01:16 Show structure disclaimer 00:04:16 Passionate discussion 00:06:26 WHy greatness cant be planned and the tyranny of objectives 00:14:40 Chinese Finger Trap 00:16:28 Perverse Incentives and feedback loops 00:18:17 Deception 00:23:29 Maze example 00:24:44 How can we define curiosity or interestingness 00:26:59 Open endedness 00:33:01 ICML 2019 and Yannic, POET, first MSLST 00:36:17 evolutionary algorithms++ 00:43:18 POET, the first MLST 00:45:39 A lesson to GOFAI people 00:48:46 Machine Learning -- the great stagnation 00:54:34 Actual scientific successes are usually luck, and against the odds -- Biontech 00:56:21 Picbreeder and NEAT 01:10:47 How Tim applies these ideas to his life and why he runs MLST 01:14:58 Keith Skit about UCF 01:15:13 Main show kick off 01:18:02 Why does Kenneth value serindipitous exploration so much 01:24:10 Scientific support for Keneths ideas in normal life 01:27:12 We should drop objectives to achieve them. An oxymoron? 01:33:13 Isnt this just resource allocation between exploration and exploitation? 01:39:06 Are objectives merely a matter of degree? 01:42:38 How do we allocate funds for treasure hunting in society 01:47:34 A keen nose for what is interesting, and voting can be dangerous 01:53:00 Committees are the antithesis of innovation 01:56:21 Does Kenneth apply these ideas to his real life? 01:59:48 Divergence vs interestingness vs novelty vs complexity 02:08:13 Picbreeder 02:12:39 Isnt everything novel in some sense? 02:16:35 Imagine if there was no selection pressure? 02:18:31 Is innovation == environment exploitation? 02:20:37 Is it possible to take shortcuts if you already knew what the innovations were? 02:21:11 Go Explore -- does the algorithm encode the stepping stones? 02:24:41 What does it mean for things to be interestingly different? 02:26:11 behavioral characterization / diversity measure to your broad interests 02:30:54 Shaping objectives 02:32:49 Why do all ambitious objectives have deception? Picbreeder analogy 02:35:59 Exploration vs Exploitation, Science vs Engineering 02:43:18 Schools of thought in ML and could search lead to AGI 02:45:49 Official ending