Multi-objective Big Data Optimization with jMetal and Spark

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10173)


Big Data Optimization is the term used to refer to optimization problems which have to manage very large amounts of data. In this paper, we focus on the parallelization of metaheuristics with the Apache Spark cluster computing system for solving multi-objective Big Data Optimization problems. Our purpose is to study the influence of accessing data stored in the Hadoop File System (HDFS) in each evaluation step of a metaheuristic and to provide a software tool to solve these kinds of problems. This tool combines the jMetal multi-objective optimization framework with Apache Spark. We have carried out experiments to measure the performance of the proposed parallel infrastructure in an environment based on virtual machines in a local cluster comprising up to 100 cores. We obtained interesting results for computational effort and propose guidelines to face multi-objective Big Data Optimization problems.


Multi-objective optimization Big Data jMetal Spark Parallel computing 



This work has been partially funded by Grants TIN2014-58304-R (Spanish Ministry of Education and Science) and P11-TIC-7529 (Innovation, Science and Enterprise Ministry of the regional government of the Junta de Andalucía) and P12-TIC-1519 (Plan Andaluz de Investigación, Desarrollo e Innovación). Cristóbal Barba-González is supported by Grant BES-2015-072209 (Spanish Ministry of Economy and Competitiveness).


  1. 1.
    Abdul-Rahman, S., Bakar, A.A., Mohamed-Hussein, Z.-A.: Optimizing big data in bioinformatics with swarm algorithms. In: IEEE 16th International Conference on Computational Science and Engineering (CSE), pp. 1091–1095, December 2013Google Scholar
  2. 2.
    Aljarah, I., Ludwig, S.A.: Mapreduce intrusion detection system based on a particle swarm optimization clustering algorithm. In: IEEE Congress on Evolutionary Computation (CEC 2013), pp. 955–962, June 2013Google Scholar
  3. 3.
    Thomas, S.A., Jin, Y.: Reconstructing biological gene regulatory networks: where optimization meets big data. Evol. Intel. 7(1), 29–47 (2014)CrossRefGoogle Scholar
  4. 4.
    Barba-González, C., Nebro, A.J., Cordero, J.A., García-Nieto, J., Durillo, J.J., Navas-Delgado, I., Aldana-Montes, J.F.: jMetalSP: a framework for dynamic multi-objective big data optimization. Applied Soft Computing (2016, submitted)Google Scholar
  5. 5.
    Cabanas-Abascal, A., García-Machicado, E., Prieto-González, L., de Amescua Seco, A.: An item based geo-recommender system inspired by artificial immune algorithms. J. Univ. Comput. Sci. 19(13), 2013–2033 (2013)Google Scholar
  6. 6.
    Coello, C., Lamont, G.B., van Veldhuizen, D.A.: Multi-objective Optimization Using Evolutionary Algorithms, 2nd edn. Wiley, New York (2007)zbMATHGoogle Scholar
  7. 7.
    Cordero, J.A., Nebro, A.J., Barba-González, C., Durillo, J.J., García-Nieto, J., Navas-Delgado, I., Aldana-Montes, J.F.: Dynamic multi-objective optimization with jmetal and spark: a case study. In: Pardalos, P.M., Conca, P., Giuffrida, G., Nicosia, G. (eds.) MOD 2016. LNCS, vol. 10122, pp. 106–117. Springer, Heidelberg (2016). doi: 10.1007/978-3-319-51469-7_9 CrossRefGoogle Scholar
  8. 8.
    Corne, D.W., Jerram, N.R., Knowles, J.D., Oates, M.J.: PESA-II: region-based selection in evolutionary multi-objective optimization. In: Genetic and Evolutionary Computation Conference (GECCO 2001), pp. 283–290. Morgan Kaufmann (2001)Google Scholar
  9. 9.
    Daoudi, M., Hamena, S., Benmounah, Z., Batouche, M.: Parallel diffrential evolution clustering algorithm based on MapReduce. In: 6th International Conference of Soft Computing and Pattern Recognition (SoCPaR 2014), pp. 337–341 (2014)Google Scholar
  10. 10.
    Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)CrossRefGoogle Scholar
  11. 11.
    Durillo, J.J., Nebro, A.J.: jMetal: a Java framework for multi-objective optimization. Adv. Eng. Softw. 42, 760–771 (2011)CrossRefGoogle Scholar
  12. 12.
    Govindarajan, K., Somasundaram, T.S., Kumar, V.S., Kinshuk: Continuous clustering in big data learning analytics. In: IEEE Fifth International Conference on Technology for Education (T4E), pp. 61–64, December 2013Google Scholar
  13. 13.
    Kitzler, E., Deb, K., Thiele, L.: Comparasion of multiobjective evolutionary algorithms: empirical results. Evol. Comput. 8(2), 173–195 (2000)CrossRefGoogle Scholar
  14. 14.
    Kukkonen, S., Lampinen, J.: GDE3: the third evolution step of generalized differential evolution. In: IEEE Congress on Evolutionary Computation (CEC 2005), pp. 443–450 (2005)Google Scholar
  15. 15.
    Lammel, R.: Google’s MapReduce programming model revisited. Sci. Comput. Program. 70(1), 1–30 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Lee, W., Hsiao, Y., Hwang, W.: Designing a parallel evolutionary algorithm for inferring gene networks on the cloud computing environment. BMC Syst. Biol. 8(1) (2014).
  17. 17.
    Luque, G., Alba, E.: Parallel Genetic Algorithms, 1st edn. Springer, Heidelberg (2011)CrossRefzbMATHGoogle Scholar
  18. 18.
    McNabb, A.W., Monson, C.K., Seppi, K.D.: Parallel PSO using MapReduce. IEEE Cong. Evol. Comput. CEC 2007, 7–14 (2007)Google Scholar
  19. 19.
    Nebro, A.J., Durillo, J.J., Vergne, M.: Redesigning the jMetal multi-objective optimization framework. In: Genetic and Evolutionary Computation Conference (GECCO 2015) Companion, pp. 1093–1100, July 2015Google Scholar
  20. 20.
    Nebro, A.J., Durillo, J.J., García-Nieto, J., Coello Coello, C.A., Luna, F., Alba, E.: SMPSO: a new PSO-based metaheuristic for multi-objective optimization. In: IEEE Symposium on Computational Intelligence in Multicriteria Decision-Making (MCDM 2009), pp. 66–73. IEEE Press (2009)Google Scholar
  21. 21.
    Shvachko, K., Kuang, H., Radia, S., Chansler R.: The Hadoop distributed file system. In: Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST 2010), Washington, DC, USA, pp. 1–10. IEEE Computer Society (2010)Google Scholar
  22. 22.
    Sun, W., Zhang, N., Wang, H., Yin, W., Qiu, T.: PACO: a period ACO based scheduling algorithm in cloud computing. In: International Conference on Cloud Computing and Big Data (CloudCom-Asia), pp. 482–486, December 2013Google Scholar
  23. 23.
    Tannahill, K.B., Jamshidi, M.: System of systems and big data analytics bridging the gap. Comput. Electr. Eng. 40(1), 2–15 (2014)CrossRefGoogle Scholar
  24. 24.
    Wu, B., Wu, G., Yang, M.: A MapReduce based ant colony optimization approach to combinatorial optimization problems. In: 8th International Conference on Natural Computation (ICNC 2012), pp. 728–732, May 2012Google Scholar
  25. 25.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S. Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud 2010, Berkeley, CA, USA, p. 10. USENIX Association (2010)Google Scholar
  26. 26.
    Zhou, Z., Chawla, N.V., Jin, Y., Williams, G.J.: Big data opportunities and challenges: Discussions from data analytics perspectives. IEEE Comput. Intell. Mag. 9(4), 62–74 (2014)CrossRefGoogle Scholar
  27. 27.
    Zitzler, E., Laumanns, M., Thiele, L.: SPEA2: improving the strength pareto evolutionary algorithm. In: Evolutionary Methods for Design, Optimization and Control with Applications to Industrial Problems, EUROGEN 2001, Greece, Athens, pp. 95–100 (2002)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Dept. de Lenguajes y Ciencias de la Computación, ETSI InformáticaUniversity of MalagaMalagaSpain

Personalised recommendations