188: World Models
Intro topic: Running News/Links: Flow matching versus diffusion models https://youtu.be/firXjwZ_6KI?is=QMq8DcCsXTTktOuE OpenCV 5 https://opencv.org/opencv-5/ Claude fable beats pokemon with no harness https://youtu.be/Ty_50J84fMY?si=EJ1KjZCZegipfCsV Can the stockmarket swallow Anthropic, SpaceX and OpenAI? https://archive.ph/nKEVw Book of the Show Patrick Strength of the Few - James Islington https://amzn.to/4pmr10T Jason Descender - Jeff Lemire https://amzn.to/3QKhp3l Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h Tool of the Show Patrick No Man’s Sky Jason Paperlib https://paperlib.app/en/ Topic: World Models Making decisions with AI Action-Value (called a Q model): What is the long-term value of making a decision at a position Policy: What action should I take (must be a distribution) Value (called a V model): What is the value of a position (depends on policy) Advantage/Disadvantage: difference in value given two policies When advantage is +, do that more. Model-Free Look at the current situation and suggest an action Run that action in the real world and measure the effect Use that measurement to suggest better actions next time Model-Based Observe rollouts (sequences of situations) and learn the dynamics Choose an action, use your dynamics model to measure the consequence Potentially do MPC (try many actions and choose the best) World Models Observe many many rollouts and learn a full forward model (how to create the input in the future) Train a policy & value inside the world model Deploy the policy and fine-tune based on the real world
