Advertisement

Constructive Cascade Learning Algorithm for Fully Connected Networks

  • Xing Wu
  • Pawel RozyckiEmail author
  • Janusz Kolbusz
  • Bogdan M. Wilamowski
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11508)

Abstract

The Fully Connected Cascade Networks (FCCN) were originally proposed along with the Cascade Correlation (CasCor) learning algorithm that having three main advantages over the Multilayer Perceptron (MLP): the structure of the network could be determined dynamically; they were more powerful for complex feature representation; the training was efficient by optimizing newly added neuron only in every stage. However, at the same time, they were criticized that the freezing strategy usually resulted in an overlarge network with the architecture much deeper than necessary. To overcome the disadvantage, in this paper, a new hybrid constructive learning (HCL) algorithm is proposed to build a FCCN as compact as possible. The proposed HCL algorithm is compared with the CasCor algorithm and some other algorithms on several popular regression benchmarks.

Keywords

Fully Connected Cascade Networks (FCCN) Hybrid Constructive Learning (HCL) algorithm Particle Swarm Optimization (PSO) Levenberg Marquardt (LM) algorithm Least Square (LS) method 

References

  1. 1.
    Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)CrossRefGoogle Scholar
  2. 2.
    Fahlman, S.E.: Fast learning variations on back-propagation: an empirical study. In: Touretzky, D., Hinton, G., Sejnowski, T. (eds.) Proceedings of the 1988 Connectionist Models Summer School (Pittsburgh, 1988), pp. 38–51. Morgan Kaufmann, San Mateo (1989)Google Scholar
  3. 3.
    Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: Ruspini, H. (ed.) Proceeding of the IEEE International Conference on Neural Networks (ICNN), San Francisco, pp. 586–591 (1993)Google Scholar
  4. 4.
    Hagan, M.T., Menhaj, M.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Networks 5, 989–993 (1994)CrossRefGoogle Scholar
  5. 5.
    Battiti, R., Masulli, F.: BFGS optimization for faster automated supervised learning. In: International Neural-Network Conference, vol. 2, pp. 757–760 (1990)Google Scholar
  6. 6.
    Kwok, T.Y., Yeung, D.Y.: Objective functions for training new hidden units in constructive neural networks. IEEE Trans. Neural Networks 8(5), 1131–1148 (1997)CrossRefGoogle Scholar
  7. 7.
    Hussain, S., Mokhtar, M., Howe, J.M.: Sensor failure detection, identification, and accommodation using fully connected cascade neural network. IEEE Trans. Industr. Electron. 62(3), 1683–1692 (2015)CrossRefGoogle Scholar
  8. 8.
    Deshpande, G., Wang, P., Rangaprakash, D., Wilamowski, B.M.: Fully connected cascade artificial neural network architecture for attention deficit hyperactivity disorder classification from functional magnetic resonance imaging data. IEEE Trans. Cybern. 45(12), 2668–2679 (2015)CrossRefGoogle Scholar
  9. 9.
    Huang, G.-B., Chen, L., Siew, C.-K.: Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Networks 17(4), 879–892 (2006)CrossRefGoogle Scholar
  10. 10.
    Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems, vol. 2, pp. 524–532. Morgan Kaufmann, San Mateo (1990)Google Scholar
  11. 11.
    Kwok, T.K., Young, D.Y.: Experimental analysis of input weight freezing in constructive neural networks. In: Proceedings of IEEE International Conference Neural Networks, San Francisco, pp. 511–516 (1993)Google Scholar
  12. 12.
    Baluja, S., Fahlman, S.: Reducing network depth in the cascade-correlation learning architecture, Technical report, Carnegie Mellon University, PittsburghGoogle Scholar
  13. 13.
    Prechelt, L.: Investigating the cascor family of learning algorithms. Neural Networks 10(5), 885–896 (1997)CrossRefGoogle Scholar
  14. 14.
    Huang, G.-B., Chen, L.: Orthogonal least squares algorithm for training cascade neural networks. IEEE Trans. Circuits Syst. I Regul. Pap. 59(11), 2629–2637 (2012)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Treadgold, N.K., Gedeon, T.D.: A cascade network algorithm employing progressive RPROP. In: Mira, J., Moreno-Díaz, R., Cabestany, J. (eds.) IWANN 1997. LNCS, vol. 1240, pp. 733–742. Springer, Heidelberg (1997).  https://doi.org/10.1007/BFb0032532CrossRefGoogle Scholar
  16. 16.
    Wu, X., Rozycki, P., Wilamowski, B.M.: Single layer feedforward networks construction based on orthogonal least square and particle swarm optimization. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9692, pp. 158–169. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-39378-0_15CrossRefGoogle Scholar
  17. 17.
    Wu, X., Rozycki, P., Wilamowski, B.M.: A hybrid constructive algorithm for single layer feedforward networks learning. IEEE Trans Neural Networks Learn. Syst. 26(8), 1659–1668 (2015)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Hwang, J.N., You, S.S., Lay, S.R., Jou, I.C.: The cascade-correlation learning: a projection pursuit learning perspective. IEEE Trans. Neural Networks 7, 278–289 (1996)CrossRefGoogle Scholar
  19. 19.
    Wilamowski, B.M., Yu, H.: Improved computation for Levenberg-Marquardt training. IEEE Trans. Neural Networks 21(6), 930–937 (2010)CrossRefGoogle Scholar
  20. 20.
    Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Networks 21(11), 1793–1803 (2010)CrossRefGoogle Scholar
  21. 21.
    Kennedy, J., Eberhart, R.C.: Particle swarm optimization. In: Proceedings of IEEE International Conference on Neural Network, Perth, Australia, pp. 1942–1948 (1995)Google Scholar
  22. 22.
    Hwang, J.N., Lay, S.R., Maechler, M., Martin, D., Schimert, J.: Regression modeling in backpropagation and projection pursuit learning. IEEE Trans. Neural Networks 5, 342–353 (1994)CrossRefGoogle Scholar
  23. 23.
    Treadgold, N.K., Gedeon, T.D.: Simulated annealing and weigh decay in adaptive learning: the SARPROP algorithm. IEEE Trans. Neural Networks 9, 662–668 (1998)CrossRefGoogle Scholar
  24. 24.
    Hunter, D., Yu, H., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures-a comparative study. IEEE Trans. Industr. Inf. 8(2), 228–240 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Xing Wu
    • 1
  • Pawel Rozycki
    • 2
    Email author
  • Janusz Kolbusz
    • 2
  • Bogdan M. Wilamowski
    • 1
  1. 1.Auburn UniversityAuburnUSA
  2. 2.University of Information Technology and Management in RzeszowRzeszowPoland

Personalised recommendations