Skip to content
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.

Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.

Podcast Webpage: https://www.astronomer.io/podcast/

Play
  • 33 episodes
  • weekly
  • Avg 25 min
  • English
Counted on this page — what you have heard stays on this device, so it is not something the list can be paged by.
  • S1 · E83
    June 11 · 22 min

    Managing a Customer Analytics Platform with Airflow at Skimlinks

    Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.  Key Takeaways 00:00 Introduction. 00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers. 02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users. 03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries. 04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management. 06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers. 08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved. 09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both. 10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration. 10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform. 14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers. 17:33 Breaking a single Python class into single-responsibility components for the DataPipe project. 19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time. 20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure. 22:34 What Julian wants from Airflow next: asset watchers paired with data contracts. Resources Mentioned Skimlinks - skimlinks.com Apache Airflow - airflow.apache.org Astronomer - astronomer.io Google BigQuery - cloud.google.com/bigquery Apache Druid - druid.apache.org Pydantic - docs.pydantic.dev Looker - cloud.google.com/looker Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

    • Transcript
  • S1 · E82
    June 4 · 21 min

    Building a custom Tableau provider for Airflow at JLR

    JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production. Key Takeaways: 00:00 Introduction. 00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands. 01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics. 03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system. 05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh. 06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished. 08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly. 08:47 Why the script approach didn't scale across teams — code duplication and version drift. 10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock. 12:21 Building a custom provider on top of the community one to keep PAT auth. 13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale. 15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility. 17:47 How the gate blocks broken DAGs from reaching production. 18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage. Resources Mentioned: JLR - jaguarlandrover.com Apache Airflow - airflow.apache.org Astronomer - astronomer.io Tableau - tableau.com Tableau REST API - help.tableau.com/current/api/rest_api/en-us/REST/rest_api.htm Airflow Tableau provider (community) - airflow.apache.org/docs/apache-airflow-providers-tableau Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

    • Transcript
  • S1 · E81
    May 28 · 32 min

    Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira

    Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data. Key Takeaways: 00:00 Introduction 01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it. 04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work. 06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat. 08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000 pipelines. 14:23 The YAML wrapper that lets non-engineers ship DAGs. Reads namespace, tables, and Spark options. Handles CDC, JDBC full, and JDBC incremental collection types with checkpoints. All changes go through data reliability engineering. 17:20 Why metadata is the most valuable asset in the AI era, and how the wrapper makes data lineage observable across 2,000 pipelines. 18:26 The Data Reliability Engineering team. A 10-person group that is the window to the company, handling maintenance, validation, corrections, and optimization for the business unit pipelines. 20:09 Operating at four petabytes of data. 21:24 Why they built custom Spark operators. Cost drove the move off the DataprocOperator. The custom operator exposes Spark driver and executor sizing as Airflow parameters and generates the Kubernetes manifest. 24:36 The monitoring dashboard built on the Airflow metadata DB. A timeline view that shows how many DAGs run each hour, used to spread scheduling across the day. 26:37 Classifying DAGs by their last five runs: success, partially correct, intermittent, total failure. A reusable observability pattern. 29:57 How to reach Mateus, and a closing thought in Portuguese on appreciating the good old times while you are living them. Resources Mentioned: Apache Airflow (airflow.apache.org) Magalu Cloud / MGC Luiza Labs (luizalabs.com) and Magazine Luiza / Magalu Astro Observe (https://www.astronomer.io/product) Mateus Ferreira on LinkedIn (linkedin.com/in/mateusmferreira) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

    • Transcript
  • S1 · E80
    May 21 · 26 min

    Enhancing DAGs for Data Processing with William Orgertrice III at Cargill

    In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform. Key Takeaways: 00:00 Introduction. 01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets. 03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations. 05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack. 06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines. 09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production. 13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder. 14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone. 15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream. 21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time. Resources Mentioned: Cargill | Website https://www.cargill.com/food-beverage Airflow Community on Slack https://airflow.apache.org/community/ Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

    • Transcript
    • Chapters
  • S1 · E79
    May 14 · 27 min

    Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer

    In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path. Shrividya Hegde,  a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong. Key Takeaways: 00:00 Introduction. 04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first. 07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries. 08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem. 13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output. 15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable. 17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code. 23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions. Resources Mentioned: Shrividya Hegde https://www.linkedin.com/in/shrividya-hegde-shri-91562365/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Women in Data | Website https://womenindata.mn.co/landing Apache Airflow Slack https://airflow.apache.org/ Shrividya's Medium writing https://medium.com/@shrihegde Shrividya’ Substack writing https://substack.com/@shrividyahegde Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

    • Transcript
    • Chapters
  • S1 · E79
    April 30 · 19 min

    Building Airflow CTL with Buğra Öztürk at Mollie

    Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.Key Takeawa...

  • S1 · E76
    April 9 · 26 min

    Introducing Airflow 3.2

    We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.Astronomer’s Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what’s new, from asset partitioning...

  • S1 · E75
    April 3 · 26 min

    Reflections on a Decade of Data Engineering at Seattle Data Guy

    Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the ...

Showing 21–33 of 33 episodes