Sign inOpen the app
Programming Throwdown

188: World Models

July 9 · 1 hr 36 min · Episode 188 · 92.2 MB
0:00-1:36:01

Streams straight from the publisher. podnod never proxies or re-hosts episode audio.

Open in podnod
Intro topic: Running

News/Links:

Book of the Show
Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h
Tool of the ShowTopic: World Models

  • Making decisions with AI
    • Action-Value (called a Q model): What is the long-term value of making a decision at a position
    • Policy: What action should I take (must be a distribution)
    • Value (called a V model): What is the value of a position (depends on policy)
    • Advantage/Disadvantage: difference in value given two policies
    • When advantage is +, do that more.
  • Model-Free
    • Look at the current situation and suggest an action
    • Run that action in the real world and measure the effect
    • Use that measurement to suggest better actions next time
  • Model-Based
    • Observe rollouts (sequences of situations) and learn the dynamics
    • Choose an action, use your dynamics model to measure the consequence
    • Potentially do MPC (try many actions and choose the best)
  • World Models
    • Observe many many rollouts and learn a full forward model (how to create the input in the future)
    • Train a policy & value inside the world model
    • Deploy the policy and fine-tune based on the real world