Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · August 6 · 27 min

Using Airflow for diverse client projects at Accion Labs

When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines. Key Takeaways: 00:00 Introduction. 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail. 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next. 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic. 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite. 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%. 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net. 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system. 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline. 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live. 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face. 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale. Resources Mentioned: [Orchestrate Everything](https://astronomer.link/data-flowcast-oe) [Accion Labs](accionlabs.com) [Apache Airflow](airflow.apache.org) [Airflow LangChain provider](airflow.apache.org/docs/apache-airflow-providers-langchain/stable/index.html) [Airflow OpenAI provider](airflow.apache.org/docs/apache-airflow-providers-openai/stable/index.html) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

0:00-27:36
Play in

transcript

No transcript — this publisher did not publish one.

show notes

When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, [Chandan Gowda](linkedin.com/in/chandan-gowda-a-h-744908195), Data Engineer at [Accion Labs](accionlabs.com), joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines.


Key Takeaways:

  • 00:00 Introduction.
  • 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail.
  • 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next.
  • 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic.
  • 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite.
  • 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%.
  • 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net.
  • 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system.
  • 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline.
  • 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live.
  • 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face.
  • 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

links5

more episodes

All episodes