Advertisement

Evaluating MapReduce on Virtual Machines: The Hadoop Case

  • Shadi Ibrahim
  • Hai Jin
  • Lu Lu
  • Li Qi
  • Song Wu
  • Xuanhua Shi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5931)

Abstract

MapReduceis emerging as an important programming model for large scale parallel application. Meanwhile, Hadoop is an open source implementation of MapReduce enjoying wide popularity for developing data intensive applications in the cloud. As, in the cloud, the computing unit is virtual machine (VM) based; it is feasible to demonstrate the applicability of MapReduce on virtualized data center. Although the potential for poor performance and heavy load no doubt exists, virtual machines can instead be used to fully utilize the system resources, ease the management of such systems, improve the reliability, and save the power. In this paper, a series of experiments are conducted to measure and analyze the performance of Hadoop on VMs. Our experiments are used as a basis for outlining several issues that will need to be considered when implementing MapReduce to fit completely in the cloud.

Keywords

Cloud Computing Data Intensive MapReduce Hadoop Distributed File System Virtual Machine 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Szalay, A., Bunn, A., Gray, J., Foster, I., Raicu, I.: The Importance of Data Locality in Distributed Computing Applications. In: Proceedings of the NSF Workflow Workshop (2006)Google Scholar
  2. 2.
    Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system. In: Proceedings of 19th ACM Symposium on Operating Systems Principles, pp. 29–43. ACM Press, New York (2003)Google Scholar
  3. 3.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: Proceedings of 6th Conference on Operating Systems Design & Implementation (2004)Google Scholar
  4. 4.
  5. 5.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for Multi-core and Multiprocessor Systems. In: Proceedings of 13th International Symposium on High Performance Computer Architecture, pp. 13–24. ACM Press, New York (2007)CrossRefGoogle Scholar
  6. 6.
    Bryant, R.E.: Data-Intensive Supercomputing: The Case for DISC. CMU-CS-07-128, Technical Report, Department of Computer Science, Carnegie Mellon University (May 2007)Google Scholar
  7. 7.
    Chen, S., Schlosser, S.W.: Map-Reduce Meets Wider Varieties of Applications, IRP-TR-08-05, Technical Report, Intel. Research Pittsburgh (May 2008)Google Scholar
  8. 8.
    CNET news, http://news.cnet.com/8301-13505_3-10196871-16.html (accessed September 2009)
  9. 9.
    Amazon Elastic Cloud Computing, http://aws.amazon.com/ec2/
  10. 10.
    GoGrid Cloud Hosting, http://www.gogrid.com/
  11. 11.
    Figueiredo, R., Dinda, P., Fortes, J.: A Case for Grid Computing on Virtual Machines. In: Proceedings of 23rd International Conference on Distributed Computing Systems, pp. 550–559. IEEE CS Press, Los Alamitos (2003)CrossRefGoogle Scholar
  12. 12.
    Mergen, M.F., Uhlig, V., Krieger, O., Xenidis, J.: Virtualization for High Performance Computing. ACM SIGOPS Oper. Syst. Rev. 40(2), 8–11 (2006)CrossRefGoogle Scholar
  13. 13.
    Huang, W., Liu, J., Abali, B., Panda, D.K.: A Case for High Performance Computing with Virtual Machines. In: Proceedings of 20th ACM International Conference on Supercomputing, pp. 125–134. ACM Press, New York (2006)CrossRefGoogle Scholar
  14. 14.
    Nagarajan, A.B., Mueller, F., Engelmann, C., Scott, S.L.: Proactive Fault Tolerance for HPC with Xen Virtualization. In: Proceedings of 21st ACM International Conference on Supercomputing, pp. 23–32. ACM Press, New York (2007)Google Scholar
  15. 15.
    Amazon Elastic MapReduce, http://aws.amazon.com/elasticmapreduce/
  16. 16.
    Amazon Simple Storage Service, http://aws.amazon.com/s3/
  17. 17.
    Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach, C., Pratt, I., Warfield, A.: Live Migration of Virtual Machines. In: Proceedings of USENIX Symposium on Networked Systems Design and Implementation (2005)Google Scholar
  18. 18.
    Zhao, M., Figueiredo, R.J.: Experimental Study of Virtual Machine Migration in Support of Reservation of Cluster Resources. In: Proceedings of 2nd International Workshop on Virtualization Technology in Distributed Computing (2007)Google Scholar
  19. 19.
    XenSource (2008), http://www.xensource.com/
  20. 20.
    Hadoop Wiki (2008), http://wiki.apache.org/hadoop/
  21. 21.
    Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: Proceedings of 8th USENIX Symposium on Operating Systems Design and Implementation (2008)Google Scholar
  22. 22.
    Ibrahim, S., Jin, H., Cheng, B., Cao, H., Wu, S., Qi, L.: Cloudlet: Towards MapReduce implementation on Virtual machines. In: Proceedings of 18th ACM International Symposium on High Performance Distributed Computing, pp. 65–66. ACM Press, New York (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Shadi Ibrahim
    • 1
  • Hai Jin
    • 1
  • Lu Lu
    • 1
  • Li Qi
    • 2
  • Song Wu
    • 1
  • Xuanhua Shi
    • 1
  1. 1.Cluster and Grid Computing Lab Services Computing Technology and System LabHuazhong University of Science & TechnologyWuhanChina
  2. 2.Operation CenterChina Development BankBeijingChina

Personalised recommendations