Advertisement

Awareness Learning for Balancing Performance and Diversity in Neural Network Ensembles

  • Yong LiuEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1074)

Abstract

An ensemble learning system requires a set of cooperative modules that solve a task together. It is believed that awareness modules being aware of other modules in an ensemble could build strong cooperations each other. In this paper, two levels of awareness are introduced into negative correlation learning in order to control the differences among the individual modules in an ensemble. At the ensemble level, whether individual modules would learn to be more different to the rest of modules on a given data point would depend on how well the data point had been learned by the ensemble. At the individual level, each individual module would try to learn to be more different to the rest of modules on a given data point only if it had not been far away from the other individuals based on the distances measured by their outputs on the data point. Negative correlation learning with such awareness learning was tested on training different structures of neural network ensembles where the relations between differences and cooperation were analyzed in the learning process.

References

  1. 1.
    Hinton, G.E., Vinyals, O., Dean, J.: Dark knowledge. Invited talk at the BayLearn Bay Area Machine Learning Symposium (2014)Google Scholar
  2. 2.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  3. 3.
    Chilimbi, T., Suzue, Y., Apacible, J., Kalyanaraman, K.: Project adam: building an efficient and scalable deep learning training system. In: 11th USENIX Symposium on Operating Systems Design and Implementation (2014)Google Scholar
  4. 4.
    Cohen, N., Sharir, O., Shashua, A.: On the expressive power of deep learning: a tensor analysis. arXiv:1509.05009 (2015)
  5. 5.
    Liu, Y., Yao, X.: Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 29(6), 716–725 (1999)CrossRefGoogle Scholar
  6. 6.
    Liu, Y.: Error awareness by lower and upper bounds in ensemble learning. Int. J. Pattern Recognit. Artif. 30(9) (2016) Google Scholar
  7. 7.
    Liu, Y.: Negative selection in negative correlation learning. In: The 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (2016)Google Scholar
  8. 8.
    Liu, Y.: Build correlation awareness in negative correlation learning. In: The 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (2017)Google Scholar
  9. 9.
    Liu, Y.: Negative correlation learning with multiple target values. In: The 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (2018)Google Scholar
  10. 10.
    Liu, Y.: Computational awareness for learning neural network ensembles. In: The 2017 IEEE International Conference on Information and Automation (2017)Google Scholar
  11. 11.
    Liu, Y., Zhao, Q., Pei, Y.: Error awareness by lower and upper bounds in ensemble learning. In: The 11th International Conference on Natural Computation (2015)Google Scholar
  12. 12.
    Liu, Y.: Control of the error signals by self-awareness in committee machines. In: The 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.School of Computer Science and EngineeringThe University of AizuAizu-WakamatsuJapan

Personalised recommendations