Advertisement

Neural Computing and Applications

, Volume 32, Issue 2, pp 313–322 | Cite as

Analysis on the potential of an EA–surrogate modelling tandem for deep learning parametrization: an example for cancer classification from medical images

  • Ruxandra StoeanEmail author
S.I. : IWANN2017: Learning algorithms with real world applications

Abstract

The paper introduces a novel modality to efficiently tune the convolutional layers of a deep neural network (CNN) and an approach to also rank the importance of the involved hyperparameters. Evolutionary algorithms (EA) offer a flexible solution to this twofold issue, while the expensive simulations of the deep learner with the generated configurations are resolved by surrogate modelling. Three models have been used and evaluated as surrogates: random forests (RF), support vector machines (SVM) and Kriging. Sample convolutional configurations are generated by Latin hypercube sampling and have attached computed accuracy outcomes from real CNN runs. For the hyperparameter estimation task, the fitness of an individual from the EA associated with a surrogate model is subsequently derived from the CNN accuracy estimation on those variable values. With respect to the ranking and variable selection task, RF includes implicit variable selection, the SVM can be straightforwardly supported by a second EA, and Kriging offers a ranking based on the corresponding θ values. The estimated accuracy of the found hyperparameter values is compared with the true validation accuracy, and they are next used for the prediction on the test cases. The ranking of the variables for each of the three surrogate models is compared, and their influence is also revealed by response surface methodology. The experimental testing of the proposed EA–surrogate approaches is conducted on a real-world scenario of histopathological image interpretation in colorectal cancer diagnosis.

Keywords

Convolutional neural network Parametrization Surrogate model Evolutionary algorithms 

Notes

Compliance with ethical standards

Conflict of Interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Albarqouni S, Baur C, Achilles F, Belagiannis V, Demirci S, Navab N (2016) Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans Med Imaging 35(5):1313–1321.  https://doi.org/10.1109/TMI.2016.2528120 CrossRefGoogle Scholar
  2. 2.
    Arunkumar R, Karthigaikumar P (2017) Multi-retinal disease classification by reduced deep learning features. Neural Comput Appl 28(2):329–334.  https://doi.org/10.1007/s00521-015-2059-9 CrossRefGoogle Scholar
  3. 3.
    Atencia M, Joya G, Sandoval F (2004) Parametric identification of robotic systems with stable time-varying hopfield networks. Neural Comput Appl 13(4):270–280.  https://doi.org/10.1007/s00521-004-0421-4 CrossRefzbMATHGoogle Scholar
  4. 4.
    Bacciu D, Lisboa PJG, Martín JD, Stoean R, Vellido A (2018) Bioinformatics and medicine in the era of deep learning. CoRR arXiv:1802.09791
  5. 5.
    Bartz-Beielstein T, Zaefferer M (2017) Model-based methods for continuous and discrete global optimization. Appl Soft Comput 55:154–167.  https://doi.org/10.1016/j.asoc.2017.01.039 CrossRefGoogle Scholar
  6. 6.
    Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. Springer, Berlin, pp 411–418Google Scholar
  7. 7.
    Eiben AE, Smith JE (2003) Introduction to evolutionary computing. Springer, BerlinCrossRefGoogle Scholar
  8. 8.
    Forrester AIJ, Sbester A, Keane AJ (2008) Engineering design via surrogate modelling: a practical guide. Wiley, New YorkCrossRefGoogle Scholar
  9. 9.
    Friese M, Bartz-Beielstein T, Emmerich M (2016) Building ensembles of surrogates by optimal convex combination. In: Papa G, Mernik M (eds) Bioinspired optimization methods and their applications. Jožef Stefan Institute, Lubljana, pp 131–143Google Scholar
  10. 10.
    Gorunescu F, Belciug S (2016) Boosting backpropagation algorithm by stimulus-sampling: application in computer-aided medical diagnosis. J Biomed Inform 63:74–81.  https://doi.org/10.1016/j.jbi.2016.08.004 CrossRefGoogle Scholar
  11. 11.
    Hackl C (2014) Calibration and parameterization methods for the libor market model. Springer, BerlinCrossRefGoogle Scholar
  12. 12.
    Heaton JB, Polson NG, Witte JH (2017) Deep learning for finance: deep portfolios. Appl Stoch Models Bus Ind 33(1):3–12.  https://doi.org/10.1002/asmb.2209 MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Jiang F, Grigorev A, Rho S, Tian Z, Fu Y, Jifara W, Adil K, Liu S (2017) Medical image semantic segmentation based on deep learning. Neural Comput Appl.  https://doi.org/10.1007/s00521-017-3158-6 CrossRefGoogle Scholar
  14. 14.
    Jin Y (2011) Surrogate-assisted evolutionary computation: recent advances and future challenges. Swarm Evolut Comput 1(2):61–70CrossRefGoogle Scholar
  15. 15.
    Field RV Jr, Constantine P, Boslough M (2013) Statistical surrogate models for prediction of high-consequence climate change. Int J Uncertain Quant 3(4):341–355MathSciNetCrossRefGoogle Scholar
  16. 16.
    Kapás Z, Lefkovits L, Iclanzan D, Gyorfi Á, Iantovics B, Lefkovits S, Szilágyi SM, Szilágyi L (2017) Automatic brain tumor segmentation in multispectral MRI volumes using a random forest approach. PSIVT 10749:137–149Google Scholar
  17. 17.
    Loshchilov I, Hutter F (2016) CMA-ES for hyperparameter optimization of deep neural networks. CoRR arXiv:1604.07269
  18. 18.
    Malshe M, Narulkar R, Raff LM, Hagan M, Bukkapatnam S, Komanduri R (2008) Parametrization of analytic interatomic potential functions using neural networks. J Chem Phys 129(4):044,111.  https://doi.org/10.1063/1.2957490 CrossRefGoogle Scholar
  19. 19.
    Martens D, Baesens B, Gestel TV (2009) Decompositional rule extraction from support vector machines by active learning. IEEE Trans Knowl Data Eng 21(2):178–191.  https://doi.org/10.1109/TKDE.2008.131 CrossRefGoogle Scholar
  20. 20.
    McKay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245MathSciNetzbMATHGoogle Scholar
  21. 21.
    Paja W, Pancerz K (2017) Feature selection methods applied to severe brain damages data. In: 2017 Federated conference on computer science and information systems (FedCSIS), pp 199–202.  https://doi.org/10.15439/2017F382
  22. 22.
    Postavaru S, Stoean R, Stoean C, Joya G (2017) Adaptation of deep convolutional neural networks for cancer grading from histopathological images. In: Advances in computational intelligence: 14th international work-conference on artificial neural networks, IWANN 2017, Cadiz, Spain, 14–16 June 2017, Proceedings, Part II, Rojas, Ignacio and Joya, Gonzalo and Catala, Andreu (eds). Springer International Publishing, Cham, pp 38–49CrossRefGoogle Scholar
  23. 23.
    Preuss M (2015) Multimodal optimization by means of evolutionary algorithms. Natural computing series. Springer, Berlin.  https://doi.org/10.1007/978-3-319-07407-8 CrossRefzbMATHGoogle Scholar
  24. 24.
    Sikora EBTST (2017) Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In: International conference on image processing. IEEE SigPort. http://sigport.org/2022
  25. 25.
    Sirinukunwattana K, Raza SEA, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35(5):1196–1206.  https://doi.org/10.1109/TMI.2016.2525803 CrossRefGoogle Scholar
  26. 26.
    Stoean C (2016) In search of the optimal set of indicators when classifying histopathological images. In: 2016 18th International symposium on symbolic and numeric algorithms for scientific computing (SYNASC), pp 449–455.  https://doi.org/10.1109/SYNASC.2016.074
  27. 27.
    Stoean C, Preuss M, Stoean R (2013) EA-based parameter tuning of multimodal optimization performance by means of different surrogate models. In: Genetic and evolutionary computation conference, GECCO 2013. ACM, pp 1063–1070.  https://doi.org/10.1145/2464576.2482684
  28. 28.
    Stoean C, Stoean R, Sandita A, Ciobanu D, Mesina C, Gruia CL (2016) SVM-based cancer grading from histopathological images using morphological and topological features of glands and nuclei. Springer, Berlin, pp 145–155.  https://doi.org/10.1007/978-3-319-39345-2_13 CrossRefGoogle Scholar
  29. 29.
    Taylor S, Kim T, Yue Y, Mahler M, Krahe J, Rodriguez AG, Hodgins J, Matthews I (2017) A deep learning approach for generalized speech animation. ACM Trans Graph 36(4):93:1–93:11.  https://doi.org/10.1145/3072959.3073699 CrossRefGoogle Scholar
  30. 30.
    Urda D, Montes-Torres J, Moreno F, Franco L, Jerez JM (2017) Deep learning to analyze RNA-Seq gene expression data. Springer International Publishing, Berlin, pp 50–59.  https://doi.org/10.1007/978-3-319-59147-6_5 CrossRefGoogle Scholar
  31. 31.
    Volz V, Rudolph G, Naujoks B (2017) Investigating uncertainty propagation in surrogate-assisted evolutionary algorithms. In: Proceedings of the genetic and evolutionary computation conference, GECCO ’17. ACM, New York, NY, USA, pp 881–888.  https://doi.org/10.1145/3071178.3071249
  32. 32.
    Yoon JG, Heo J, Kim M, Park YJ, Choi MH, Song J, Wyi K, Kim H, Duchenne O, Eom S, Tsoy Y (2018) Machine learning-based diagnosis for disseminated intravascular coagulation (DIC): development, external validation, and comparison to scoring systems. PLOS ONE 13(5):1–15.  https://doi.org/10.1371/journal.pone.0195861 CrossRefGoogle Scholar
  33. 33.
    Young SR, Rose DC, Karnowski TP, Lim SH, Patton RM (2015) Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: Proceedings of the workshop on machine learning in high-performance computing environments. ACM, pp 4:1–4:5.  https://doi.org/10.1145/2834892.2834896
  34. 34.
    Zhang J, Chowdhury S, Messac A (2012) An adaptive hybrid surrogate model. Struct Multidiscip Optim 46(2):223–238.  https://doi.org/10.1007/s00158-012-0764-x CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2018

Authors and Affiliations

  1. 1.Department of Computer Science, Faculty of SciencesUniversity of CraiovaCraiovaRomania
  2. 2.CraiovaRomania

Personalised recommendations