Apache Airflow

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Orchestrating workflows as code

Airflow works best with workflows that are mostly static and slowly changing. When a DAG (Directed Acyclic Graph) structure remains similar across execution runs, it clarifies the unit of work and continuity. Typical users are data engineers, machine learning engineers, and backend developers who need to coordinate complex sequences of tasks.

An Airflow user inputs a Python definition of a DAG. The action occurs when the Airflow scheduler executes tasks on workers while following the specified dependencies. The observable outcome is a completed pipeline whose progress, execution logs, and historical runs can be monitored and troubleshooted through Airflow's user interface. Rich command line utilities also make performing complex surgeries on DAGs straightforward.

Prerequisites and execution constraints

Running Airflow typically requires a POSIX-compliant operating system. The main development versions are tested with Python 3.10 through 3.14 and PostgreSQL versions 14 through 18. A relational database is required to store metadata, and MySQL (versions 8.0, 8.4, and Innovation) is also tested and supported. While SQLite can be used for local development tests, it is not recommended for production environments.

Airflow requires tasks to ideally be idempotent, meaning that the results of the task will be the same and will not create duplicated data in a destination system. Airflow also discourages passing large quantities of data between tasks. While tasks can share small amounts of metadata using the XCom feature, high-volume data processing should be delegated to external specialized services. Furthermore, Airflow is not a streaming solution, but processes real-time data by pulling it off streams in batches.

Installing development dependencies

When setting up a local development environment on macOS (Mojave or Catalina), the official installation documentation notes that you may need to install the Xcode command line tools along with specific database packages. The following command installs the required SQLite, MySQL, and PostgreSQL packages using Homebrew:

Airflow leverages the Jinja templating engine to allow customizations, providing flexibility to parameterize tasks at runtime. Additionally, the Airflow framework includes a wide range of built-in operators and can be extended to fit your specific needs. Beyond traditional data pipelines, Airflow is widely used to orchestrate machine learning workflows—such as training, retraining, evaluation, and deployment—and increasingly agentic and LLM-based workloads, coordinating the steps of an AI pipeline rather than acting as the agent itself. For full project resources, see the Airflow GitHub repository.