Smart Systems_Icon_1500px

The Complete Hands-On Introduction to Apache Airflow

Enquiry
Programme Code D233
Domain
Smart Systems
Level
Foundation
Learning Partner(s)
Civil Service College
Duration
Format E-learning
Rating
Competencies
Simulation Integration and Management
Job Roles
ICT&SS Professional Specialised Systems Engineer (Digital Twin) Specialised Systems Engineer (Geospatial) Specialised Systems Engineer (IoT) Specialised Systems Engineer (M&S) Systems Engineer

Overview

Explore Apache Airflow, an open-source platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have platform as a support. Gain an in-depth learning session to understand what Airflow is, and how it works and deep dive into the advanced concepts - such as learning how to create plugins and make real dynamic pipelines.

Key Takeaways

At the end of this programme, you will be able to:
  • Create plugins to add functionalities to Apache Airflow
  • Use Docker with Airflow and different executors
  • Master core functionalities such as DAGs, Operators, Tasks, Workflows and etc
  • Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs
  • Identify the difference between Sequential, Local and Celery Executors, understand how do they work and how you can use them
  • Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc
  • Install and configure Apache Airflow
  • Think, answer and implement solutions using Airflow to real data processing problems

Who Should Attend

Please refer to the job roles section.

Prerequisites

Some prior programming or scripting experience. Python experience will help but not a must.

Programme Structure

This programme will cover the following topics:

  • Getting Started with Airflow
  • The important views of the Airflow UI
  • Coding your first data pipeline with Airflow
  • The new way of scheduling
  • Databases and Executors
  • Implementing advanced concepts in airflow
  • Creating Airflow Plugins with Elasticsearch and PostgreSQL

Refer to the programme page for further details on the content provider(s).

NOTE
This self-paced e-learning course is accessible by all Public Service Officers via this link.

This programme is complimentary.

Learners are required to register via the LEARN Portal

Step-by-Step Guide

1. Login to LEARN Portal 

2. If using laptop:

  • Type module title, e.g. The Complete Hands-On, in the search bar.
  • Click on title.
  • Click on "Start Learning" to proceed.

3. If using mobile:

  • Tap on the “Discover” tab.
  • Type module title, e.g. The Complete Hands-On, in the search bar.
  • Tap on title.
  • Tap on "Start Learning" to proceed.

Note that the process for accessing CSC LEARN modules is generic.