Overview
Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have platform as a support. Gain an in-depth learning session to understand what Airflow is, how it works and deep dive into the advanced concepts - such as learning how to create plugins and make real dynamic pipelines.
Note: This self-paced e-learning course is accessible by all Public Service Officers via
this link.
Key Takeaways
By the end of the programme, you will be able to:
- create plugins to add functionalities to Apache Airflow
- use Docker with Airflow and different executors
- master core functionalities such as DAGs, Operators, Tasks, Workflows and etc
- understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs
- the difference between Sequential, Local and Celery Executors, how do they work and how you can use them
- use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc
- install and configure Apache Airflow
- think, answer and implement solutions using Airflow to real data processing problems
Who Should Attend
- Please refer to the job role section
Prerequisites
Some prior programming or scripting experience. Python experience will help but not a must.
This programme will cover the following topics:
- Getting Started with Airflow
- The important views of the Airflow UI
- Coding your first data pipeline with Airflow
- The new way of scheduling
- Databases and Executors
- Implementing advanced concepts in airflow
- Creating Airflow Plugins with Elasticsearch and PostgreSQL
Refer to the programme page for further details on the content provider(s).
Learners are required to register via the
LEARN Portal
Step-by-step guide
Login to LEARN Portal
- If using laptop:
- Type module title, e.g. The Complete Hands-On, in the search bar.
- Click on title.
- Click on ‘Start Learning’ to proceed.
- If using mobile:
- Tap on the “Discover” tab.
- Type module title, e.g. The Complete Hands-On, in the search bar.
- Tap on title.
- Tap on ‘Start Learning’ to proceed.
Note that the process for accessing CSC LEARN modules is generic