Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · July 16 · 25 min

Orchestrating Retail Data Pipelines at Saks Global

Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security. Key Takeaways: 00:00 Introduction. 01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow. 02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems. 04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google. 07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro. 09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout. 13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend. 18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code. 21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams. Resources Mentioned: [Apache Airflow](airflow.apache.org) [Astro](astronomer.io/product) [Otto, the Astronomer data engineering agent](astronomer.io) [Snowflake](snowflake.com) [Saks Global](saks.com) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

0:00-25:20
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security.


Key Takeaways:

  • 00:00 Introduction.
  • 01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow.
  • 02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems.
  • 04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google.
  • 07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro.
  • 09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout.
  • 13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend.
  • 18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code.
  • 21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

links6

more episodes

All episodes