Skip to content
Sign in
Artwork for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI · June 25 · 18 min

Running Airflow 3 in a regulated environment at OTPP

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution. Key Takeaways: 00:00 Introduction. 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement. 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience. 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle. 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC. 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode. 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly. 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster. 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way. 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production. 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording. 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution. Resources Mentioned: [Ontario Teachers' Pension Plan](otpp.com) [Apache Airflow](airflow.apache.org) [Astronomer](astronomer.io) [Cosmos](astronomer.io/cosmos) Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow

0:00-18:49
Play in

transcript

No transcript — this publisher did not publish one.

show notes

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.


Key Takeaways:

  • 00:00 Introduction.
  • 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
  • 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
  • 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
  • 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
  • 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
  • 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
  • 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
  • 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
  • 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
  • 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
  • 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.


Resources Mentioned:


Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.


#AI #Automation #Airflow

links4

more episodes

All episodes