Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · July 23 · 22 min

Orchestrating AI video intelligence and evaluation pipelines at Firework

How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Airflow project. Key Takeaways: 00:00 Introduction. 00:48 What Firework does: a video commerce platform bringing short-form shoppable video and livestream experiences directly onto brand websites and apps. 02:13 Shawn's team owns the full data and AI platform stack at Firework, from ingestion through BI and applied AI. 02:59 Airflow has been the main orchestration layer for batch transformations since the early open source days, and now powers AI workflows like conversational insights, knowledge-base building, and automated eval pipelines. 06:45 The AI workflow stack: ingestion of user interaction and content data, transformation into personalized profiles and embeddings, separate eval pipelines dispatched from third-party systems, all on Snowflake with Cortex AI and coordinated by Airflow. 08:55 Why Airflow stuck: flexibility, reliability, and clear visibility into dependencies across SQL, Python, and AI tasks. 11:27 The deterministic pipeline problem. Airflow assumes predictable input and output. LLM workflows break that assumption, which forces evaluation layers, traceability, regression testing, and feedback loops into the pipeline itself. 13:07 Evaluation cost can exceed compute cost. Validating a single output can mean running four or five parallel eval jobs across prompts and configurations, so caching, batching, and selective evaluation become essential. 15:58 Building safeguards: alerting and aborting jobs when cost exceeds thresholds, so the platform stays operationally stable while other improvements catch up. 18:30 Wishlist: stronger first-class support for AI workflow patterns (evaluation tracking, prompt experimentation, model observability, event-driven AI orchestration) and continued UI/UX improvement in Airflow 3. Resources Mentioned: [Firework](firework.com) [Apache Airflow](airflow.apache.org) [Snowflake Cortex AI](snowflake.com/en/data-cloud/cortex) [Airflow common.ai provider](airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html) Shawn Feng on LinkedIn (https://www.linkedin.com/in/shawnshifeng/) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #ApacheAirflow

0:00-22:31
Play in

transcript

No transcript — this publisher did not publish one.

show notes

How do you orchestrate AI workflows when LLM outputs are non-deterministic and evaluation costs can quietly exceed compute costs? Shawn Feng, Head of Data at [Firework](firework.com), joins the show to walk through how his team uses Airflow to coordinate ingestion, embeddings, and evaluation pipelines for AI features in a video commerce platform. The conversation covers the technical stack, the challenges of validating LLM output, cost guardrails, and what Shawn wants to see next from the Airflow project.


Key Takeaways:

  • 00:00 Introduction.
  • 00:48 What Firework does: a video commerce platform bringing short-form shoppable video and livestream experiences directly onto brand websites and apps.
  • 02:13 Shawn's team owns the full data and AI platform stack at Firework, from ingestion through BI and applied AI.
  • 02:59 Airflow has been the main orchestration layer for batch transformations since the early open source days, and now powers AI workflows like conversational insights, knowledge-base building, and automated eval pipelines.
  • 06:45 The AI workflow stack: ingestion of user interaction and content data, transformation into personalized profiles and embeddings, separate eval pipelines dispatched from third-party systems, all on Snowflake with Cortex AI and coordinated by Airflow.
  • 08:55 Why Airflow stuck: flexibility, reliability, and clear visibility into dependencies across SQL, Python, and AI tasks.
  • 11:27 The deterministic pipeline problem. Airflow assumes predictable input and output. LLM workflows break that assumption, which forces evaluation layers, traceability, regression testing, and feedback loops into the pipeline itself.
  • 13:07 Evaluation cost can exceed compute cost. Validating a single output can mean running four or five parallel eval jobs across prompts and configurations, so caching, batching, and selective evaluation become essential.
  • 15:58 Building safeguards: alerting and aborting jobs when cost exceeds thresholds, so the platform stays operationally stable while other improvements catch up.
  • 18:30 Wishlist: stronger first-class support for AI workflow patterns (evaluation tracking, prompt experimentation, model observability, event-driven AI orchestration) and continued UI/UX improvement in Airflow 3.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #ApacheAirflow

links5

more episodes

All episodes