Workflow Orchestration with Apache Airflow

Abstract

Generally speaking, there are two kinds of problems you’ll find yourself running into more often than not as a data engineer. The first stems from broken promises, aka bad upstream data sources, and the more general realm of the unknown unknowns with respect to data movement through your data pipelines. The second problem you’ll find yourself up against is time. This is not the part in the book where I start to talk to you about life, death, and decision making, but rather time as a boundary or a threshold. Time exists between the physical runtime of jobs, as well as a very real line in the sand when it relates to data service level agreements (SLAs). These data contracts revolve around expectations in terms of the data format (aka schemas) as well as the agreed upon time when data should be expected to become available. Another way in which time gets the best of us is at the intersection of both of these common problems, e.g., upstream problems married happily with stale data, or missed SLAs.