ECJ+HADOOP: An Easy Way to Deploy Massive Runs of Evolutionary Algorithms

  • Francisco ChávezEmail author
  • Francisco Fernández
  • César Benavides
  • Daniel Lanza
  • Juan Villegas
  • Leonardo Trujillo
  • Gustavo Olague
  • Graciela Román
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9598)


This paper describes initial steps towards allowing Evolutionary Algorithms (EAs) researchers to easily deploy computing intensive runs of EAs on Big Data infrastructures. Although many proposals have already been described in the literature, and a number of new software tools have been implemented embodying parallel versions of EAs, we present here a different approach. Given traditional resistance to change when adopting new software, we try instead to endow the well known ECJ tool with the MapReduce model. By using the Hadoop framework, we introduce changes in ECJ that allow researchers to launch any EA problem on a big data infrastructure similarly as when a single computer is used to run the algorithm. By means of a new parameter, researchers can choose where the run will be launched, whether in a Hadoop based infrastructure or in a desktop computer. This paper shows the tests performed, how the whole system has been tuned to optimize the running time for ECJ experiments, and finally a realworld problem is shown to describe how the MapReduce model can automatically deploy the tasks generated by ECJ without additional intervention.


Multi-objective evolutionary algorithm Face recognition Hadoop ECJ 



This work has been supported by FP7-PEOPLE-2013-IRSES, Grant 612689 ACoBSEC, Spanish Ministry of Economy, Project UEX:EPHEMEC (TIN2014-56494-C4-2-P); Junta de Extremadura, and FEDER, project GR15068. Also it has been supported by CONACyT México by the project 155045 – “Evolución de Cerebros Artificiales en Visión por Computadora” and TESE by the project DIMI-MCIM-004/08.


  1. 1.
    González, A., Prieto, F.: Extracción de puntos característicos del rostro para medidas antropométricas. Revista Ingenierías Universidad de Medellín, 9(17), 139–150 (2010)Google Scholar
  2. 2.
    Jain, A.K.: Automatic face recognition: state of the art, Distinguished Lecture Series, 0–44, Septiembre (2010)Google Scholar
  3. 3.
    Sun, K., Kang, H., Park, H.-H.: Tagging and classifying facial images in cloud environments based on KNN using MapReduce. Optik Int. J. Light Electron Opt. 126(21), 3227–3233 (2015). ISSN 0030–4026. Google Scholar
  4. 4.
    Zhang, Z., Li, W., Jia, H.: A fast face recognition algorithm based on MapReduce. In: 2014 Seventh International Symposium on Computational Intelligence and Design (ISCID), vol. 2, pp. 395–399, 13–14 December 2014. doi: 10.1109/ISCID.2014.195
  5. 5.
    Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: a literature survey. ACM Comput. Surv. (CSUR) 35(4), 399–458 (2003)CrossRefGoogle Scholar
  6. 6.
    Yang, M.H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)CrossRefGoogle Scholar
  7. 7.
    Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)CrossRefGoogle Scholar
  8. 8.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886, 16–21 June 2012. doi: 10.1109/CVPR.2012.6248014
  9. 9.
    Zhang, L., Yang, M., Feng, X.: Sparse representation or collaborative representation: which helps face recognition? In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 471–478, 6–13 November 2011. doi: 10.1109/ICCV.2011.6126277
  10. 10.
    Wagner, A., Wright, J., Ganesh, A., Zhou, Z., Mobahi, H., Ma, Y.: Toward a practical face recognition system: robust alignment and illumination by sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 34(2), 372–386 (2012). doi: 10.1109/TPAMI.2011.112 CrossRefGoogle Scholar
  11. 11.
    Fernández, F., Sánchez, J.M., Tomassini, M., Gómez, J.A.: A parallel genetic programming tool based on PVM. In: Margalef, T., Dongarra, J., Luque, E. (eds.) PVM/MPI 1999. LNCS, vol. 1697, pp. 241–248. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  12. 12.
    Tomassini, M., Vanneschi, L., Bucher, L., Fernandez de Vega, F.: An MPI-based tool for distributed genetic programming. In: IEEE International Conference on Cluster Computing (CLUSTER 2000) pp. 209–209. IEEE Computer Society (2013)Google Scholar
  13. 13.
    Shvachko, K., Hairong, K., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10 (2010)Google Scholar
  14. 14.
    White, T.: Hadoop: the definitive guide (2009)Google Scholar
  15. 15.
  16. 16.
    ECJ: A Java-based Evolutionary Computation Research System.
  17. 17.
    Dean, J., Ghemawat, S.: MapReduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRefGoogle Scholar
  18. 18.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  19. 19.
    Cantu-Paz, E.: Efficient and Accurate Parallel Genetic Algorithms. Genetic Algorithms and Evolutionary Computation, vol. 1. Springer, New York (2011)zbMATHGoogle Scholar
  20. 20.
    Fernandez, F., Tomassini, M., Vanneschi, L.: An empirical study of multipopulation genetic programming. Genet. Program. Evolvable Mach. 4(1), 21–51Google Scholar
  21. 21.
    Melab, N., Cahon, S., Talbi, E.G.: Grid computing for parallel bioinspired algorithms. J. Parallel Distrib. Comput. 66(8), 1052–1061Google Scholar
  22. 22.
    Gonzalez, D.L., et al.: Increasing gp computing power for free via desktop grid computing and virtualization. In: 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pp. 419–423. IEEE (2009)Google Scholar
  23. 23.
    García-Valdez, M., Trujillo, L., Fernández de Vega, F., Merelo Guervós, J.J., Olague, G.: EvoSpace: a distributed evolutionary platform based on the tuple space model. In: Esparcia-Alcázar, A.I. (ed.) EvoApplications 2013. LNCS, vol. 7835, pp. 499–508. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  24. 24.
    Sherry, D., Veeramachaneni, K., McDermott, J., O’Reilly, U.-M.: Flex-GP: genetic programming on the cloud. In: Di Chio, C., et al. (eds.) EvoApplications 2012. LNCS, vol. 7248, pp. 477–486. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  25. 25.
    Fernndez de Vega, F., Chvez, F., Trujillo, L., Mediero, E., Muoz, L.: A Hybrid ECJ+Boinc tool for distributed evolutionary algorithms. In: Research in Computing Science, pp. 120–130 (2014)Google Scholar
  26. 26.
    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection, vol. 1. MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  27. 27.
    Laredo, J.L.J., Eiben, A.E., Steen, M., Merelo, J.J.: Evag: a scalable peer-to-peer evolutionary algorithm. Genet. Program. Evolvable Mach. 11(2), 227–246 (2010)CrossRefGoogle Scholar
  28. 28.
    Du, X., Ni, Y., Yao, Z., Xiao, R., Xie, D.: High Performance parallel evolutionary algorithm model based on mapreduce framework. Int. J. Comput. Appl. Technol. 46(1), 290–295 (2013)CrossRefGoogle Scholar
  29. 29.
    Stelzer, D., Mellis, W.: Success factors of organizational change in software process improvement. Softw. Process Improv. Pract. 4(4), 227–250 (1998)CrossRefGoogle Scholar
  30. 30.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106(1), 59–70 (2007)CrossRefGoogle Scholar
  31. 31.
    Benavide, C., Villegas, J., Román, G., Avilés, C.: Face recognition using CBIR techniques (Spanish). In: Proceedings MAEB 2015, pp. 733–740 (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Francisco Chávez
    • 1
    Email author
  • Francisco Fernández
    • 1
  • César Benavides
    • 2
  • Daniel Lanza
    • 3
  • Juan Villegas
    • 4
  • Leonardo Trujillo
    • 5
  • Gustavo Olague
    • 6
  • Graciela Román
    • 2
  1. 1.Department of Computer ScienceUniversity of ExtremaduraMéridaSpain
  2. 2.Departamento de Ing. EléctricaUniversidad Autónoma Metropolitana, IztapalapaMexico, D.F.Mexico
  3. 3.CERN, European Organization for Nuclear ResearchMeyrinSwitzerland
  4. 4.Departamento de ElectrónicaUniversidad Autónoma Metropolitana, AzcapotzalcoMexico, D.F.Mexico
  5. 5.Instituto Tecnológico de TijuanaTijuanaMexico
  6. 6.CICESEEnsenadaMexico

Personalised recommendations