Journal of Grid Computing

, Volume 16, Issue 2, pp 265–284 | Cite as

Migration-Aware Genetic Optimization for MapReduce Scheduling and Replica Placement in Hadoop

  • Carlos Guerrero
  • Isaac Lera
  • Carlos Juiz


This work addresses the optimization of file locality, file availability, and replica migration cost in a Hadoop architecture. Our optimization algorithm is based on the Non-dominated Sorting Genetic Algorithm-II and it simultaneously determines file block placement, with a variable replication factor, and MapReduce job scheduling. Our proposal has been tested with experiments that considered three data center sizes (8, 16 and 32 nodes) with the same workload and number of files (150 files and 3519 file blocks). In general terms, the use of a placement policy with a variable replica factor obtains higher improvements for our three optimization objectives. On the contrary, the use of a job scheduling policy only improves these objectives when it is used along a variable replication factor. The results have also shown that the migration cost is a suitable optimization objective as significant improvements up to 34% have been observed between the experiments.


Resource management Genetic algorithm Multi-objective optimization Replica placement MapReduce scheduling Hadoop 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



This research was supported by Ministerio de Economía, Industria y Competitividad (MINECO) of Spain and the European Commission (FEDER funds) throught the grant number TIN2017-88547-P.


  1. 1.
    Beloglazov, A., Buyya, R.: Energy efficient allocation of virtual machines in cloud data centers. In: 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, pp 577–578 (2010),
  2. 2.
    Borthakur, D., et al.: Hdfs architecture guide. Hadoop Apache Project 53 (2008)Google Scholar
  3. 3.
    Bose, S.K., Brock, S., Skeoch, R., Rao, S.: Cloudspider: combining replication with scheduling for optimizing live migration of virtual machines across wide area networks. In: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID ’11, pp 13–22. IEEE Computer Society, Washington, DC (2011),
  4. 4.
    Bryk, P., Malawski, M., Juve, G., Deelman, E.: Storage-aware algorithms for scheduling of workflow ensembles in clouds. J. Grid Comput. 14(2), 359–378 (2016). CrossRefGoogle Scholar
  5. 5.
    Chen, Y., Ganapathi, A., Griffith, R., Katz, R.: The case for evaluating mapreduce performance using workload suites. In: 2011 IEEE 19th Annual International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp 390–399 (2011),
  6. 6.
    Cheng, Z., Luan, Z., Meng, Y., Xu, Y., Qian, D., Roy, A., Zhang, N., Guan, G.: Erms: an elastic replication management system for hdfs. In: 2012 IEEE International Conference on Cluster Computing Workshops, pp 32–40 (2012),
  7. 7.
    Dai, W., Ibrahim, I., Bassiouni, M.: A new replica placement policy for hadoop distributed file system. In: 2016 IEEE 2nd International Conference on Big Data Security on Cloud (BigDataSecurity), IEEE International Conference on High Performance and Smart Computing (HPSC), and IEEE International Conference on Intelligent Data and Security (IDS), pp 262–267 (2016),
  8. 8.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. In: Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation - Volume 6, OSDI’04, pp 10–10. USENIX Association, Berkeley (2004).
  9. 9.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: Nsga-ii. Trans. Evol. Comput. 6(2), 182–197 (2002). CrossRefGoogle Scholar
  10. 10.
    Durillo, J.J., Prodan, R.: Multi-objective workflow scheduling in amazon ec2. Cluster Comput. 17(2), 169–189 (2014). CrossRefGoogle Scholar
  11. 11.
    Eltabakh, M.Y., Tian, Y., Özcan, F., Gemulla, R., Krettek, A., McPherson, J.: Cohadoop: Flexible data placement and its exploitation in hadoop. Proc. VLDB Endow. 4(9), 575–585 (2011). CrossRefGoogle Scholar
  12. 12.
    Ghomi, E.J., Rahmani, A.M., Qader, N.N.: Load-balancing algorithms in cloud computing: a survey. J. Netw. Comput. Appl. 88, 50–71 (2017). []CrossRefGoogle Scholar
  13. 13.
    Grace, R.K., Manimegalai, R.: Dynamic replica placement and selection strategies in data grids—a comprehensive survey. J. Parallel Distrib. Comput. 74 (2), 2099–2108 (2014). []CrossRefGoogle Scholar
  14. 14.
    Guerrero, C., Lera, I., Juiz, C.: Genetic algorithm for multi-objective optimization of container allocation in cloud architecture. J. Grid Comput. (2017)
  15. 15.
    Guzek, M., Bouvry, P., Talbi, E.G.: A survey of evolutionary computation for resource management of processing in cloud computing [review article]. IEEE Comput. Intell. Mag. 10(2), 53–67 (2015). CrossRefGoogle Scholar
  16. 16.
    Hamrouni, T., Slimani, S., Charrada, F.B.: A survey of dynamic replication and replica selection strategies based on data mining techniques in data grids. Eng. Appl. Artif. Intell. 48, 140–158 (2016). []CrossRefGoogle Scholar
  17. 17.
    Hashem, I.A.T., Anuar, N.B., Marjani, M., Gani, A., Sangaiah, A.K., Sakariyah, A.K.: Multi-objective scheduling of mapreduce jobs in big data processing. Multimed. Tools Appl. 1–16. (2017)
  18. 18.
    Hashem, I.A.T., Yaqoob, I., Anuar, N.B., Mokhtar, S., Gani, A., Khan, S.U.: The rise of “big data” on cloud computing: review and open research issues. Inf. Syst. 47, 98–115 (2015). []CrossRefGoogle Scholar
  19. 19.
    Ibn-Khedher, H., Hadji, M., Abd-Elrahman, E., Afifi, H., Kamal, A.E.: Scalable and cost efficient algorithms for virtual cdn migration. In: 2016 IEEE 41st Conference on Local Computer Networks (LCN), pp 112–120 (2016),
  20. 20.
    Khezr, S.N., Navimipour, N.J.: Mapreduce and its applications, challenges, and architecture: a comprehensive review and directions for future research. J. Grid Comput. 15(3), 295–321 (2017). CrossRefGoogle Scholar
  21. 21.
    Kimovski, D., Saurabh, N., Stankovski, V., Prodan, R.: Multi-objective middleware for distributed VMI repositories in federated cloud environment. Scalable Comput.: Pract. Exp. 17(4), 299–312 (2016) []Google Scholar
  22. 22.
    Lammel, R.: Google’s mapreduce programming model. revisited. Sci. Comput. Program. 70(1), 1–30 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Long, S.Q., Zhao, Y.L., Chen, W.: Morm: a multi-objective optimized replication management strategy for cloud storage cluster. J. Syst. Archit. 60(2), 234–244 (2014). []CrossRefGoogle Scholar
  24. 24.
    López-Pires, F., Barán, B.: Many-objective virtual machine placement. J. Grid Comput. 15 (2), 161–176 (2017). CrossRefGoogle Scholar
  25. 25.
    Lu, L., Shi, X., Jin, H., Wang, Q., Yuan, D., Wu, S.: Morpho: a decoupled mapreduce framework for elastic cloud computing. Futur. Gener. Comput. Syst. 36 (Supplement C), 80–90 (2014). Special Section: Intelligent Big Data Processing Special Section: Behavior Data Security Issues in Network Information Propagation Special Section: Energy-efficiency in Large Distributed Computing Architectures Special Section: eScience Infrastructure and ApplicationsCrossRefGoogle Scholar
  26. 26.
    Maheshwari, N., Nanduri, R., Varma, V.: Dynamic energy efficient data placement and cluster reconfiguration algorithm for mapreduce framework. Futur. Gener. Comput. Syst. 28(1), 119–127 (2012). []CrossRefGoogle Scholar
  27. 27.
    Maio, V.D., Prodan, R., Benedict, S., Kecskemeti, G.: Modelling energy consumption of network transfers and virtual machine migration. Futur. Gener. Comput. Syst. 56, 388–406 (2016). []CrossRefGoogle Scholar
  28. 28.
    Malik, S.U.R., Khan, S.U., Ewen, S.J., Tziritas, N., Kolodziej, J., Zomaya, A.Y., Madani, S.A., Min-Allah, N., Wang, L., Xu, C.Z., Malluhi, Q.M., Pecero, J.E., Balaji, P., Vishnu, A., Ranjan, R., Zeadally, S., Li, H.: Performance analysis of data intensive cloud systems based on data management and replication: a survey. Distrib. Parallel Databases 34(2), 179–215 (2016). CrossRefGoogle Scholar
  29. 29.
    Mansouri, Y., Toosi, A.N., Buyya, R.: Cost optimization for dynamic replication and migration of data in cloud data centers. IEEE Trans. Cloud Comput. PP(99), 1–1 (2017). CrossRefGoogle Scholar
  30. 30.
    Marler, R.T., Arora, J.S.: The weighted sum method for multi-objective optimization: new insights. Struct. Multidiscip. Optim. 41(6), 853–862 (2010). MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Marozzo, F., Talia, D., Trunfio, P.: P2p-mapreduce: parallel data processing in dynamic cloud environments. J. Comput. Syst. Sci. 78(5), 1382–1402 (2012). JCSS Special Issue: Cloud Computing 2011CrossRefGoogle Scholar
  32. 32.
    Milani, B.A., Navimipour, N.J.: A comprehensive review of the data replication techniques in the cloud environments: major trends and future directions. J. Netw. Comput. Appl. 64, 229–238 (2016). []CrossRefGoogle Scholar
  33. 33.
    Pawlikowski, K.: Steady-state simulation of queueing processes: Survey of problems and solutions. ACM Comput. Surv. 22 (2), 123–170 (1990). []CrossRefGoogle Scholar
  34. 34.
    Semenkin, E., Semenkina, M.: Self-configuring Genetic Algorithm with Modified Uniform Crossover Operator, pp 414–421. Berlin, Heidelberg (2012)Google Scholar
  35. 35.
    Shen, H., Sarker, A., Yu, L., Deng, F.: Probabilistic network-aware task placement for mapreduce scheduling. In: 2016 IEEE International Conference on Cluster Computing (CLUSTER), pp 241–250 (2016),
  36. 36.
    Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp 1–10 (2010),
  37. 37.
    Song, J., He, H., Wang, Z., Yu, G., Pierson, J.M.: Modulo based data placement algorithm for energy consumption optimization of mapreduce system. J. Grid Comput. (2016)
  38. 38.
    Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., Baldeschwieler, E.: Apache hadoop yarn: Yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing, SOCC ’13, pp 5:1–5:16. ACM, New York (2013),
  39. 39.
    Wang, F., Qiu, J., Yang, J., Dong, B., Li, X., Li, Y.: Hadoop high availability through metadata replication. In: Proceedings of the First International Workshop on Cloud Data Management, CloudDB ’09, pp 37–44. ACM, New York (2009),
  40. 40.
    Wang, W., Zhu, K., Ying, L., Tan, J., Zhang, L.: Maptask scheduling in mapreduce with data locality: throughput and heavy-traffic optimality. IEEE/ACM Trans. Netw. 24 (1), 190–203 (2016). CrossRefGoogle Scholar
  41. 41.
    Wang, X., Wang, Y., Cui, Y.: A new multi-objective bi-level programming model for energy and locality aware multi-job scheduling in cloud computing. Futur. Gener. Comput. Syst. 36, 91–101 (2014). Special Section: Intelligent Big Data ProcessingSpecial Section: Behavior Data Security Issues in Network Information PropagationSpecial Section: Energy-efficiency in Large Distributed Computing Architectures Special Section: eScience Infrastructure and ApplicationsCrossRefGoogle Scholar
  42. 42.
    Wei, G., Vasilakos, A.V., Zheng, Y., Xiong, N.: A game-theoretic method of fair resource allocation for cloud computing services. J. Supercomput. 54(2), 252–269 (2010). CrossRefGoogle Scholar
  43. 43.
    Wei, Q., Veeravalli, B., Gong, B., Zeng, L., Feng, D.: Cdrm: a cost-effective dynamic replication management scheme for cloud storage cluster. In: 2010 IEEE International Conference on Cluster Computing, pp. 188–196 (2010),
  44. 44.
    Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. Trans. Evol. Comput. 1(1), 67–82 (1997). CrossRefGoogle Scholar
  45. 45.
    Wu, J., Yuan, H., He, Y., Zou, Z.: Chordmr: a p2p-based job management scheme in cloud. J. Netw. 9, 541–548 (2014)Google Scholar
  46. 46.
    Xie, T., Sun, Y.: A file assignment strategy independent of workload characteristic assumptions. Trans. Storage 5 (3), 10:1–10:24 (2009). []MathSciNetCrossRefGoogle Scholar
  47. 47.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10, pp 10–10. USENIX Association, Berkeley (2010).
  48. 48.
    Zhan, Z.H., Liu, X.F., Gong, Y.J., Zhang, J., Chung, H.S.H., Li, Y.: Cloud computing resource scheduling and a survey of its evolutionary approaches. ACM Comput. Surv. 47(4), 63:1–63:33 (2015). []CrossRefGoogle Scholar
  49. 49.
    Zhang, Q., Pan, X., Shen, Y., Li, W.: A novel scalable architecture of cloud storage system for small files based on p2p. In: 2012 IEEE International Conference on Cluster Computing Workshops, pp 41–47 (2012),

Copyright information

© Springer Science+Business Media B.V., part of Springer Nature 2018

Authors and Affiliations

  1. 1.Computer Science DepartmentUniversity of Balearic IslandsPalmaSpain

Personalised recommendations