Effect of Live Migration on Virtual Hadoop Cluster

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10722)

Abstract

Emerging computational requirement for large scale data analysis has resulted in the importance of big data processing. Meanwhile, with virtualization it is now feasible to deploy Hadoop in private or public cloud environment which offers unique benefits like scalability, high availability etc. Live migration is an important feature provided by virtualization that migrate a running VM from one physical host to another to facilitate load balancing, maintenance, server consolidation and avoid SLA violation of VM. However, live migration adds overhead and degrades the performance of the application running inside the VM. This paper discusses the performance of Hadoop when VMs are migrated from one host to another. Experiment shows that job completion time, average downtime as well as average migration time gets increased with increase in the number of VMs that are migrated.

Keywords

Virtualization SAN Live migration Hadoop MapReduce Pre-copy 

References

  1. 1.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  2. 2.
    Borthakur, D.: The hadoop distributed file system: architecture and design. Hadoop Proj. Website 11(2007), 21 (2007)Google Scholar
  3. 3.
    Ibrahim, S., Jin, H., Lu, L., Qi, L., Wu, S., Shi, X.: Evaluating MapReduce on virtual machines: the Hadoop case. In: Jaatun, M.G., Zhao, G., Rong, C. (eds.) CloudCom 2009. LNCS, vol. 5931, pp. 519–528. Springer, Heidelberg (2009).  https://doi.org/10.1007/978-3-642-10665-1_47 CrossRefGoogle Scholar
  4. 4.
    Xu, G., Xu, F., Ma, H.: Deploying and researching hadoop in virtual machines. In: International Conference on Automation and Logistics (ICAL), 2012 IEEE, pp. 395–399. IEEE (2012)Google Scholar
  5. 5.
    Ye, K., Jiang, X., Chen, S., Huang, D., Wang, B.: Analyzing and modeling the performance in xen-based virtual cluster environment. In: 12th IEEE International Conference on High Performance Computing and Communications (HPCC), 2010, pp. 273–280. IEEE (2010)Google Scholar
  6. 6.
    Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach, C., Pratt, I., Warfield, A.: Live migration of virtual machines. In: Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation, vol. 2, pp. 273–286. USENIX Association (2005)Google Scholar
  7. 7.
    Hwang, J., Zeng, S., Wu, F.Y., Wood, T.: A component-based performance comparison of four hypervisors. In: IFIP/IEEE International Symposium on Integrated Network Management (IM 2013), 2013, pp. 269–276. IEEE (2013)Google Scholar
  8. 8.
    Johnson, C., Chiu, D.: Hadoop in flight: Migrating live mapreduce jobs for power-shifting data centers. In: IEEE 9th International Conference on Cloud Computing (CLOUD), 2016, pp. 92–99. IEEE (2016)Google Scholar
  9. 9.
    Kambatla, K., Pathak, A., Pucha, H.: Towards optimizing hadoop provisioning in the cloud. HotCloud 9, 12 (2009)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Motilal Nehru National Institute of Technology AllahabadAllahabadIndia

Personalised recommendations