Scheduling MapReduce Jobs and Data Shuffle on Unrelated Processors

Abstract

We propose a constant approximation algorithm for generalizations of the Flexible Flow Shop (FFS) problem which form a realistic model for non-preemptive scheduling in MapReduce systems. Our results concern the minimization of the total weighted completion time of a set of MapReduce jobs on unrelated processors and improve substantially on the model proposed by Moseley et al. (SPAA 2011) in two directions: (i) we consider jobs consisting of multiple Map and Reduce tasks, which is the key idea behind MapReduce computations, and (ii) we introduce into our model the crucial cost of the data shuffle phase, i.e., the cost for the transmission of intermediate data from Map to Reduce tasks. Moreover, we experimentally evaluate our algorithm compared with a lower bound on the optimal cost of our problem as well as with a fast algorithm, which combines a simple online assignment of tasks to processors with a standard scheduling policy. As we observe, for random instances that capture data locality issues, our algorithm achieves a better performance.

This research was supported by the projects “Handling uncertainty in data intensive applications on a distributed computing environment (cloud computing) (DELUGE)” (D. Fotakis, I. Milis and E. Zampetakis) and “Energy Efficiency of Road Networks and Vehicles: Measurement, Pricing, Regional and Environmental Effects (EERNV)” (G.Zois), co-financed by the European Union (European Social Fund - ESF) and Greek national funds, through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF) - Research Funding Program: THALES, investing in knowledge society through the European Social Fund. A short extended abstract of this work, including partial results, appeared in EDBT/ICDT 2014 Workshop on Algorithms for MapReduce and Beyond.