Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · Thursday · 24 min

Orchestrating strictly sequential ETL pipelines at Synechron

Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3. Key Takeaways: (00:00) Introduction. (02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports. (05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem. (06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution. (09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side. (10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs. (12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it. (15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers. (17:38) How the layers talk to each other: sensors and TriggerDagRunOperator. (19:44) Migrating from subdags to task groups without losing the ability to retry a whole group. (21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier. (23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability. Resources Mentioned: Orchestrate Everything Apache Airflow Snowflake MongoDB Apache Spark Elasticsearch Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

0:00-24:28
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.


Key Takeaways:

(00:00) Introduction.

(02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.

(05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.

(06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.

(09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.

(10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.

(12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.

(15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.

(17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.

(19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.

(21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.

(23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow



links8

more episodes

All episodes