Apache Airflow is an open up-resource job for scheduling and handling workflows, composed in Python.
Kaxil Naik, director of Airflow engineering at Astronomer and one of the main committers of Airflow, instructed SD Situations: “It is used to automate your daily careers or day-to-day tasks, and responsibilities can be as basic as functioning a Python script or it can be as intricate as bringing in all the information from 500 unique information warehouses and manipulating it.”
It was developed at Airbnb in 2014 and is about to rejoice its 10 yr anniversary later this calendar year. It joined the Apache Program Foundation in March 2016 at the Incubation amount and was produced a top rated-level undertaking in 2019.
Airflow was at first intended for just ETL use situations, but has in excess of the several years progressed to incorporate functions that make it useful for all elements linked to data engineering.
“It has ongoing to be the chief in this house, because we have maintained a superior harmony involving innovation and steadiness. Mainly because of this practically 10 several years of Airflow in the exact same room, we have included so several attributes that allow for Airflow to be really responsible and stable,” he mentioned.
The most the latest release, 2.9, arrived out previously this week and extra new options like the skill to mix dataset and time-primarily based schedules, custom names for Dynamic Task Mapping, and the skill to team undertaking logs.
The task can be located on GitHub below.
The publish SD Periods Open-Source Project of the 7 days: Apache Airflow appeared initial on SD Instances.