Advertisement

Artificial Neural Networks Training Acceleration Through Network Science Strategies

  • Lucia CavallaroEmail author
  • Ovidiu Bagdasar
  • Pasquale De Meo
  • Giacomo Fiumara
  • Antonio Liotta
Conference paper
  • 44 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11974)

Abstract

Deep Learning opened artificial intelligence to an unprecedented number of new applications. A critical success factor is the ability to train deeper neural networks, striving for stable and accurate models. This translates into Artificial Neural Networks (ANN) that become unmanageable as the number of features increases. The novelty of our approach is to employ Network Science strategies to tackle the complexity of the actual ANNs at each epoch of the training process. The work presented herein originates in our earlier publications, where we explored the acceleration effects obtained by enforcing, in turn, scale freeness, small worldness, and sparsity during the ANN training process. The efficiency of our approach has also been recently confirmed by independent researchers, who managed to train a million-node ANN on non-specialized laptops. Encouraged by these results, we have now moved into having a closer look at some tunable parameters of our previous approach to pursue a further acceleration effect. We now investigate on the revise fraction parameter, to verify the necessity of the role of its double-check. Our method is independent of specific machine learning algorithms or datasets, since we operate merely on the topology of the ANNs. We demonstrate that the revise phase can be avoided in order to half the overall execution time with an almost negligible loss of quality.

Keywords

Network Science Artificial Neural Networks 

Notes

Acknowledgments

We thank Dr Decebal Costantin Mocanu for providing constructive feedback.

References

  1. 1.
    Barabási, A.L., Pósfai, M.: Network Science. Cambridge University Press, Cambridge (2016). http://barabasi.com/networksciencebook/Google Scholar
  2. 2.
    Berman, D.S., Buczak, A., Chavis, J., Corbett, C.: A survey of deep learning methods for cyber security. Information 10, 122 (2019).  https://doi.org/10.3390/info10040122CrossRefGoogle Scholar
  3. 3.
    Cao, C., et al.: Deep learning and its applications in biomedicine. Genomics Proteomics Bioinform. 16(1), 17–32 (2018).  https://doi.org/10.1016/j.gpb.2017.07.003MathSciNetCrossRefGoogle Scholar
  4. 4.
    Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., Blaschke, T.: The rise of deep learning in drug discovery. Drug Discov. Today 23(6), 1241–1250 (2018).  https://doi.org/10.1016/j.drudis.2018.01.039CrossRefGoogle Scholar
  5. 5.
    Erdös, P., Rényi, A.: On random graphs I. Publ. Math. Debr. 6, 290–297 (1959)zbMATHGoogle Scholar
  6. 6.
    Gardner, M., Dorling, S.: Artificial neural networks (the multilayer perceptron)–a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)CrossRefGoogle Scholar
  7. 7.
    Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.orgzbMATHGoogle Scholar
  8. 8.
    Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012).  https://doi.org/10.1109/MSP.2012.2205597CrossRefGoogle Scholar
  9. 9.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017).  https://doi.org/10.1145/3065386CrossRefGoogle Scholar
  10. 10.
    Latora, V., Nicosia, V., Russo, G.: Complex Networks: Principles, Methods and Applications. Cambridge University Press, Cambridge (2017)CrossRefGoogle Scholar
  11. 11.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nat. Cell Biol. 521(7553), 436–444 (2015).  https://doi.org/10.1038/nature14539CrossRefGoogle Scholar
  12. 12.
    Liu, S., Mocanu, D.C., Matavalam, A., Pei, Y., Pechenizkiy, M.: Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. arXiv:1901.09181 (2019)
  13. 13.
    Mocanu, D.C., Mocanu, E., Stone, P., Nguyen, P., Gibescu, M., Liotta, A.: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nat. Commun. 9, 2383 (2018).  https://doi.org/10.1038/s41467-018-04316-3CrossRefGoogle Scholar
  14. 14.
    Ruano-Ordás, D., Yevseyeva, I., Fernandes, V.B., Méndez, J.R., Emmerich, M.T.M.: Improving the drug discovery process by using multiple classifier systems. Expert Syst. Appl. 121, 292–303 (2019).  https://doi.org/10.1016/j.eswa.2018.12.032CrossRefGoogle Scholar
  15. 15.
    Yu, D., Deng, L.: Deep learning and its applications to signal and information processing [exploratory DSP]. IEEE Signal Process. Mag. 28(1), 145–154 (2011).  https://doi.org/10.1109/MSP.2010.939038CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of DerbyDerbyUK
  2. 2.Polo Universitario AnnunziataUniversity of MessinaMessinaItaly
  3. 3.MIFT DepartmentUniversity of MessinaMessinaItaly
  4. 4.Edinburgh Napier UniversityEdinburghUK

Personalised recommendations