Skip to content
Sign in
Artwork for Data Engineering Concepts
Data Engineering Concepts · Today · 29 min

Apache Airflow

Hi, my name is Dmitrii Polovinkin, I'm a data engineer with ~7 years of experience (check out ⁠⁠my LinkedIn⁠⁠) and welcome to the first episode of Data Engineering Concepts, which is a new educational podcast about data engineering, designed to be useful when you're learning on the go. In this episode, I explain what Apache Airflow is, what problems it solves, and when it makes sense to use it in data engineering. We go through the main Airflow concepts, including DAGs and DAG runs, tasks and task instances, operators, sensors, hooks, connections, variables, scheduling, data intervals, XComs, and trigger rules. I also give a high-level overview of Airflow architecture, executors, workers, the metadata database, and some of the key DAG configuration parameters.

0:00-29:15
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Hi, my name is Dmitrii Polovinkin, I'm a data engineer with ~7 years of experience (check out ⁠⁠my LinkedIn⁠⁠) and welcome to the first episode of Data Engineering Concepts, which is a new educational podcast about data engineering, designed to be useful when you're learning on the go.

In this episode, I explain what Apache Airflow is, what problems it solves, and when it makes sense to use it in data engineering.

We go through the main Airflow concepts, including DAGs and DAG runs, tasks and task instances, operators, sensors, hooks, connections, variables, scheduling, data intervals, XComs, and trigger rules. I also give a high-level overview of Airflow architecture, executors, workers, the metadata database, and some of the key DAG configuration parameters.

links1

more episodes

All episodes