MS

The Complete Hands-On Introduction to Apache Airflow

Enquiry
Programme Code D233
Domain
Modelling & Simulation
Level
Foundation
Learning Partner(s)
Civil Service College
Duration
Format E-learning
Rating
Competencies
Simulation Integration
Job Roles
ICT&SS Professional Modelling & Simulation Engineer

Overview

Apache Airflow is an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have platform as a support. Gain an in-depth learning session to understand what Airflow is, how it works and deep dive into the advanced concepts - such as learning how to create plugins and make real dynamic pipelines.

Note: This self-paced e-learning course is accessible by all Public Service Officers via this link.


Key Takeaways

By the end of the programme, you will be able to:

  • create plugins to add functionalities to Apache Airflow
  • use Docker with Airflow and different executors
  • master core functionalities such as DAGs, Operators, Tasks, Workflows and etc
  • understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs
  • the difference between Sequential, Local and Celery Executors, how do they work and how you can use them
  • use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc
  • install and configure Apache Airflow
  • think, answer and implement solutions using Airflow to real data processing problems


Who Should Attend

  • Please refer to the job role section 


Prerequisites

Some prior programming or scripting experience. Python experience will help but not a must.

This programme will cover the following topics:

  • Getting Started with Airflow
  • The important views of the Airflow UI
  • Coding your first data pipeline with Airflow
  • The new way of scheduling
  • Databases and Executors
  • Implementing advanced concepts in airflow
  • Creating Airflow Plugins with Elasticsearch and PostgreSQL

Refer to the programme page for further details on the content provider(s).

Coplimentary.

Learners are required to register via the LEARN Portal

Step-by-step guide
  • Login to LEARN Portal 
    1. If using laptop:
      • Type module title, e.g. The Complete Hands-On, in the search bar.
      • Click on title.
      • Click on ‘Start Learning’ to proceed.
    2. If using mobile:
      • Tap on the “Discover” tab.
      • Type module title, e.g. The Complete Hands-On, in the search bar.
      • Tap on title.
      • Tap on ‘Start Learning’ to proceed.

    Note that the process for accessing CSC LEARN modules is generic