Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · July 9 · 27 min

What's New in Apache Airflow® 3.3

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off. Key Takeaways: 00:00 Introduction. 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry. 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run. 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action. 06:58 Why XComs don't solve this problem: they get reinitialized on every retry. 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403. 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase. 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs. 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator. 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run. 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting. Resources Mentioned: Astronomer Airflow 3.3 Release Notes The Task State Store Updates to the asset partitions feature Retry policies Multi-language support 3.3 Webinar Upgrading from Airflow 2 to 3 ebook Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

0:00-27:27
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.


Key Takeaways:

  • 00:00 Introduction.
  • 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
  • 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
  • 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
  • 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
  • 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
  • 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
  • 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
  • 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
  • 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
  • 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

links9

more episodes

All episodes