Scheduling MapReduce Jobs Under Multi-round Precedences

  • D. Fotakis
  • I. Milis
  • O. Papadigenopoulos
  • V. Vassalos
  • G. ZoisEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9833)


We consider non-preemptive scheduling of MapReduce jobs consisitng of multiple map-reduce rounds so as to minimize the average weighted completion time on identical and unrelated processors. For identical processors, we present LP-based O(1)-approximation algorithms, while for unrelated processors the approximation ratio naturally depends on the maximum number of rounds of any job (a small constant in practice). For the single-round case, we substantially improve on previously best known approximation ratios for both identical and unrelated processors. Moreover, we conduct an experimental analysis and compare the performance of our algorithms against a fast heuristic and a lower bound on the optimal solution, thus demonstrating their promising practical performance.


  1. 1.
    Afrati, F.N., Das Sarma, A., Salihoglu, S., Ullman, J.D.: Upper and lower bounds on the cost of a MapReduce computation. VLDB 6(4), 277–288 (2013)Google Scholar
  2. 2.
    Afrati, F., Joglekar, M., Salihoglu, C.R.S., Ullman, J.D.: GYM: A multiround join algorithm in MapReduce (2014). arXiv:1410.4156
  3. 3.
    Aspnes, J., Azar, Y., Fiat, A., Plotkin, S., Waarts, O.: On-line routing of virtual circuits with applications to load balancing and machine scheduling. JACM 44(3), 486–504 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Chen, F., Kodialam, M.S., Lakshman, T.V.: Joint scheduling of processing and shuffle phases in mapreduce systems. In: INFOCOM, pp. 1143–1151 (2012)Google Scholar
  5. 5.
    Correa, J.R., Skutella, M., Verschae, J.: The power of preemption on unrelated machines and applications to scheduling orders. Math. Oper. Res. 37(2), 379–398 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: Simplified data processing on large clusters. In: OSDI, pp. 137–150 (2004)Google Scholar
  7. 7.
    Eastman, W.L., Even, S., Iaacs, I.M.: Bounds for the optimal scheduling of \(n\) jobs on \(m\) processors. Manage. Sci. 11, 268–279 (1964)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Fotakis, D., Milis, I., Papadigenopoulos, O., Zampetakis, E., Zois, G.: Scheduling MapReduce jobs and data shuffle on unrelated processors. In: Bampis, E. (ed.) SEA 2015. LNCS, vol. 9125, pp. 137–150. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  9. 9.
    Graham, R.L.: Bounds on multiprocessing timing anomalies. SIAP 17(2), 416–429 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Hall, L.A., Schulz, A.S., Shmoys, D.B., Wein, J.: Scheduling to minimize average completion time: Off-line and on-line approximation algorithms. MOR 22, 513–544 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Hariri, A.M., Potts, C.N.: Heuristics for scheduling unrelated parallel machines. Comp. and Oper. Res. 18(3), 323–331 (1991)CrossRefzbMATHGoogle Scholar
  12. 12.
    Karloff, H., Suri, S., Vassilvitskii, S.: A model of computation for MapReduce. In: SODA, pp. 263-285 (2010)Google Scholar
  13. 13.
    Kumar, R., Moseley, B., Vassilvitskii, S., Vattani, A.: Fast greedy algorithms in mapreduce and streaming. In: SPAA, pp. 1–10 (2013)Google Scholar
  14. 14.
    Mastrolilli, M., Svensson, O.: Hardness of approximating flow and job shop scheduling problems. JACM 58(5), 20 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Moseley, B., Dasgupta, A., Kumar, R., Sarlós, T.: On scheduling in Map-Reduce and flow-shops. In: SPAA, pp. 289–298 (2011)Google Scholar
  16. 16.
    Queyranne, M., Schulz, A.S.: Approximation bounds for a general class of precedence constrained parallel machine scheduling problems. SICOMP 35(5), 1241–1253 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Queyranne, M.: Structure of a simple scheduling polyhedron. Math. Program. 58(1), 263–285 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Shmoys, D.B., Tardos, É.: An approximation algorithm for the generalized assignment problem. Math. Program. 62, 461–474 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Yoo, D.-J., Sim, K.M.: A comparative review of job scheduling for MapReduce. In: CCIS, pp. 353–358 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • D. Fotakis
    • 1
  • I. Milis
    • 2
  • O. Papadigenopoulos
    • 1
  • V. Vassalos
    • 2
  • G. Zois
    • 2
    Email author
  1. 1.School of Electrical and Computer EngineeringNational Technical University of AthensAthensGreece
  2. 2.Department of InformaticsAthens University of Economics and BusinessAthensGreece

Personalised recommendations