The Efficiency of MapReduce in Parallel External Memory

  • Gero Greiner
  • Riko Jacob
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7256)


Since its introduction in 2004, the MapReduce framework has become one of the standard approaches in massive distributed and parallel computation. In contrast to its intensive use in practise, theoretical footing is still limited and only little work has been done yet to put MapReduce on a par with the major computational models. Following pioneer work that relates the MapReduce framework with PRAM and BSP in their macroscopic structure, we focus on the functionality provided by the framework itself, considered in the parallel external memory model (PEM). In this, we present upper and lower bounds on the parallel I/O-complexity that are matching up to constant factors for the shuffle step. The shuffle step is the single communication phase where all information of one MapReduce invocation gets transferred from map workers to reduce workers. Hence, we move the focus towards the internal communication step in contrast to previous work. The results we obtain further carry over to the BSP* model. On the one hand, this shows how much complexity can be “hidden” for an algorithm expressed in MapReduce compared to PEM. On the other hand, our results bound the worst-case performance loss of the MapReduce approach in terms of I/O-efficiency.


External Memory Internal Memory Matrix Vector Multiplication MapReduce Framework MapReduce Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aggarwal, A., Vitter, J.S.: The input/output complexity of sorting and related problems. Communications of the ACM 31(9), 1116–1127 (1988)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Arge, L., Goodrich, M.T., Nelson, M., Sitchinava, N.: Fundamental parallel algorithms for private-cache chip multiprocessors. In: Proceedings of SPAA 2008, pp. 197–206. ACM (2008)Google Scholar
  3. 3.
    Bäumker, A., Dittrich, W., auf der Heide, F.M.: Truly Efficient Parallel Algorithms: C-optimal Multisearch for an Extension of the BSP Model (extended abstract). In: Spirakis, P.G. (ed.) ESA 1995. LNCS, vol. 979, pp. 17–30. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  4. 4.
    Bender, M.A., Brodal, G.S., Fagerberg, R., Jacob, R., Vicari, E.: Optimal sparse matrix dense vector multiplication in the I/O-model. TCS 47, 934–962 (2010)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Brodal, G.S., Fagerberg, R.: On the limits of cache-obliviousness. In: Proceedings of STOC 2003, pp. 307–315. ACM, New York (2003) ISBN:1-58113-674-9Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings OSDI 2004, pp. 137–150 (2004)Google Scholar
  7. 7.
    Dean, J., Ghemawat, S.: MapReduce: A flexible data processing tool. Communications of the ACM 53(1), 72–77 (2010)CrossRefGoogle Scholar
  8. 8.
    Feldman, J., Muthukrishnan, S., Sidiropoulos, A., Stein, C., Svitkina, Z.: On distributing symmetric streaming computations. ACM Trans. Algorithms 6, 1–19 (2010)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Goodrich, M.T., Sitchinava, N., Zhang, Q.: Sorting, searching, and simulation in the mapreduce framework. CoRR, abs/1101.1902 (2011)Google Scholar
  10. 10.
    Greiner, G., Jacob, R.: Evaluating Non-square Sparse Bilinear Forms on Multiple Vector Pairs in the I/O-Model. In: Hliněný, P., Kučera, A. (eds.) MFCS 2010. LNCS, vol. 6281, pp. 393–404. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Karloff, H., Suri, S., Vassilvitskii, S.: A model of computation for MapReduce. In: Proceedings of SODA 2010, pp. 938–948. SIAM (2010)Google Scholar
  12. 12.
    Pavlo, A., Paulson, E., Rasin, A., Abadi, D.J., DeWitt, D.J., Madden, S., Stonebraker, M.: A comparison of approaches to large-scale data analysis. In: Proceedings of SIGMOD 2009, pp. 165–178. ACM (2009)Google Scholar
  13. 13.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: Proceedings of HPCA 2007, pp. 13–24. IEEE (February 2007)Google Scholar
  14. 14.
    Stonebraker, M., Abadi, D., DeWitt, D.J., Madden, S., Paulson, E., Pavlo, A., Rasin, A.: MapReduce and parallel DBMSs: Friends or foes? Communications of the ACM 53(1), 64–71 (2010)CrossRefGoogle Scholar
  15. 15.
    Valiant, L.G.: A bridging model for parallel computation. Communications of the ACM 33(8), 103–111 (1990)CrossRefGoogle Scholar
  16. 16.
    White, T.: Hadoop: The Definitive Guide, 1st edn. O’Reilly (June 2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Gero Greiner
    • 1
  • Riko Jacob
    • 1
  1. 1.Institute of Theoretical Computer ScienceETH ZurichSwitzerland

Personalised recommendations