The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.

Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.

Podcast Webpage: https://www.astronomer.io/podcast/

Play

Open in

22 episodes
weekly
Avg 24 min
English

Episodes 22

Newest Oldest Longest

All20 Unplayed20 In progress0 Played0

Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.

S1 · E92
Thursday · 28 min
Managing financial datasets with Airflow at Wise
Financial data pipelines have to be right the first time. On this episode, Kenten sits down with [Antonello Benedetto](linkedin.com/in/anbento4), Staff Data Engineer at Wise, to talk about how the central data and analytics engineering team runs Ai...
S1 · E91
August 20 · 28 min
Managing travel platform data with Airflow at Headout
Running a travel platform means dealing with fast-moving inventory, real-time fraud detection, and heavy performance marketing attribution, all while keeping data trustworthy for a data-driven org. In this episode, Mrinalini Singh, Data Platform En...
S1 · E90
August 13 · 19 min
How Airflow orchestration decisions impact Spark performance
Airflow and Spark are one of the most common combinations in modern data platforms, but the orchestration decisions made on the Airflow side often determine whether Spark jobs run fast and cheap or slow and expensive. In this episode, Kenten is joi...
S1 · E89
August 6 · 27 min
Using Airflow for diverse client projects at Accion Labs
When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to ...
S1 · E88
July 30 · 31 min
Consolidating legacy job schedulers with Airflow at Trading Technologies
Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid. In this episode, Marc Lamberti is joined by [Sanket Patel](linkedin.com/in/sanket-patel-a2a57b3a), Director of Engineering at Trading Technologies, t...
S1 · E87
July 23 · 22 min
Orchestrating AI video intelligence and evaluation pipelines at Firework
How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coor...
S1 · E86
July 16 · 25 min
Orchestrating Retail Data Pipelines at Saks Global
Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Archite...
S1 · E85
July 9 · 27 min
What's New in Apache Airflow® 3.3
Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episod...
S1 · E84
June 25 · 18 min
Running Airflow 3 in a regulated environment at OTPP
Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension P...
S1 · E83
June 11 · 22 min
Managing a Customer Analytics Platform with Airflow at Skimlinks
Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the mi...
S1 · E82
June 4 · 21 min
Building a custom Tableau provider for Airflow at JLR
JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, suppl...
S1 · E81
May 28 · 32 min
Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira
Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Da...
S1 · E80
May 21 · 26 min
Enhancing DAGs for Data Processing with William Orgertrice III at Cargill
In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG des...
S1 · E79
May 14 · 27 min
Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer
In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.Shrividya Hegde, a data and AI engineer and an Airflow champion in...
S1 · E80
May 7 · 24 min
Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development
We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.Filip Kunčar, Platform Director at ShipMonk Product Developmen...
S1 · E79
April 30 · 19 min
Building Airflow CTL with Buğra Öztürk at Mollie
Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.Key Takeawa...
S1 · E78
April 23 · 28 min
Introducing Airflow’s Common AI Provider with Pavan Kumar Gopidesu and Kaxil Naik
In this episode, we explore the newly released Apache Airflow common AI provider — what problem it solves, how it was built and what's coming next.Kaxil Naik, Senior Director of Engineering at Astronomer and Apache Airflow PMC member, and Pavan Kum...
S1 · E77
April 16 · 22 min
Building AI Debugging Agents Into Airflow DAGs at Jeppesen ForeFlight with Samantha Blaney Cuevas
Aviation data pipelines run on strict 28-day publication cycles, and the margin for error is zero. In this episode, we're joined by Samantha Blaney Cuevas, Software Engineer at Jeppesen ForeFlight, to explore how her team orchestrates a complex, ti...
S1 · E76
April 9 · 26 min
Introducing Airflow 3.2
We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning...
S1 · E75
April 3 · 26 min
Reflections on a Decade of Data Engineering at Seattle Data Guy
Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the ...

Showing 1–20 of 22 episodes