185: Workflow Orchestrators

Intro topic: Asymmetric Returns News/Links: NanoChat by Andrej Karpathy https://github.com/karpathy/nanochat Pydantic AI https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/ 1000th Starlink this year https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/ ChatGPT Apps SDK https://openai.com/index/introducing-apps-in-chatgpt/ Book of the Show Patrick The Will of the Many by James Islington https://amzn.to/43IfU8Q Jason Interview with DHH (Founder of Ruby on Rails) https://www.youtube.com/watch?v=vagyIcmIGOQ Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h Tool of the Show Patrick Factorio https://www.factorio.com/ Jason nip.io Topic: Workflow Orchestrators Why Batch jobs (embarrassingly parallel) Long-running tasks (e.g. transcoding video) Checkpointing/resuming How Message Queues Containerization Worker Pools & Autoscaling History & Backfill Steps to run workflows: Containerize the workflow definition and send to the cloud Containerize all the individual tasks Submit job(s) Examples Airflow Legacy but dominant Dagster Great UX for python developers Temporal: https://temporal.io/ The new hotness Ray Low-level but very powerful Kubeflow Designed for ML workflows, integrated dashboard ★ Support this podcast on Patreon ★