Abstract
Traditional real-valued neural networks can suppress neural inputs by setting the weights to zero or overshadow other inputs by using extreme weight values. Large network weights are undesirable because they may cause network instability and lead to exploding gradients. To penalize such large weights, adequate regularization is typically required. This work presents a feed-forward and convolutional layer architecture that constrains weights along the unit circle such that neural connections can never be eliminated or suppressed by weights, ensuring that no incoming information is lost by dying neurons. The neural network’s decision boundaries are redefined by expressing model weights as angles of phase rotations and layer inputs as amplitude modulations, with trainable weights always remaining within a fixed range. The approach can be quickly and readily integrated into existing layers while preserving the model architecture of the original network. The classification performance was tested and assessed on basic computer vision data sets using ShuffleNetv2, ResNet18, and GoogLeNet at high learning rates.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahlfors, L.V.: Complex analysis: an introduction to the theory of analytic functions of one complex variable. New York, London 177 (1953)
Aizenberg, I.: Complex-Valued Neural Networks with Multi-valued Neurons, vol. 353. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20353-4
Aizenberg, I.: The multi-valued neuron. In: Aizenberg, I. (ed.) Complex-Valued Neural Networks with Multi-valued Neurons. SCI, vol. 353, pp. 55–94. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20353-4_2
Aizenberg, I., Moraga, C.: Multilayer feedforward neural network based on multi-valued neurons (MLMVN) and a backpropagation learning algorithm. Soft. Comput. 11(2), 169–183 (2007)
Aizenberg, I., Moraga, C., Paliy, D.: A feedforward neural network based on multi-valued neurons. In: Reusch, B. (eds.) Computational Intelligence, Theory and Applications. ASC, vol. 33, pp. 599–612. Springer, Heidelberg (2005). https://doi.org/10.1007/3-540-31182-3_55
Aizenberg, I., Paliy, D., Astola, J.T.: Multilayer neural network based on multi-valued neurons and the blur identification problem. In: The 2006 IEEE International Joint Conference on Neural Network Proceedings, pp. 473–480. IEEE (2006)
Aizenberg, I., Sheremetov, L., Villa-Vargas, L., Martinez-Muñoz, J.: Multilayer neural network with multi-valued neurons in time series forecasting of oil production. Neurocomputing 175, 980–989 (2016)
Aizenberg, N.N., Aizenberg, I.N.: Cnn based on multi-valued neuron as a model of associative memory for grey scale images. In: CNNA’92 Proceedings Second International Workshop on Cellular Neural Networks and Their Applications, pp. 36–41. IEEE (1992)
Amin, M.F., Amin, M.I., Al-Nuaimi, A.Y.H., Murase, K.: Wirtinger calculus based gradient descent and Levenberg-Marquardt learning algorithms in complex-valued neural networks. In: Lu, B.-L., Zhang, L., Kwok, J. (eds.) ICONIP 2011. LNCS, vol. 7062, pp. 550–559. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24955-6_66
Amin, M.F., Murase, K.: Single-layered complex-valued neural network for real-valued classification problems. Neurocomputing 72(4–6), 945–955 (2009)
Bassey, J., Qian, L., Li, X.: A survey of complex-valued neural networks. arXiv preprint arXiv:2101.12249 (2021)
Benvenuto, N., Piazza, F.: On the complex backpropagation algorithm. IEEE Trans. Sig. Process. 40(4), 967–969 (1992)
Clarke, T.L.: Generalization of neural networks to the complex plane. In: 1990 IJCNN International Joint Conference on Neural Networks, pp. 435–440. IEEE (1990)
Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT Press, Cambridge (2016)
Hirose, A.: Dynamics of fully complex-valued neural networks. Electron. Lett. 28(16), 1492–1494 (1992)
Hirose, A.: Complex-Valued Neural Networks: Theories and Applications, vol. 5. World Scientific, Singapore (2003)
Hirose, A.: Complex-Valued Neural Networks, vol. 400. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27632-3
Hirose, A.: Complex-Valued Neural Networks: Advances and Applications, vol. 18. Wiley, Hoboken (2013)
Kuroe, Y., Yoshid, M., Mori, T.: On activation functions for complex-valued neural networks—existence of energy functions—. In: Kaynak, O., Alpaydin, E., Oja, E., Xu, L. (eds.) ICANN/ICONIP -2003. LNCS, vol. 2714, pp. 985–992. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-44989-2_117
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Leung, H., Haykin, S.: The complex backpropagation algorithm. IEEE Trans. Sig. Process. 39(9), 2101–2104 (1991)
Little, G.R., Gustafson, S.C., Senn, R.A.: Generalization of the backpropagation neural network learning algorithm to permit complex weights. Appl. Opt. 29(11), 1591–1592 (1990)
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., Alsaadi, F.E.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Lupea, V.M.: Multi-valued neuron with a periodic activation function-new learning strategy. In 2012 IEEE 8th International Conference on Intelligent Computer Communication and Processing, pp. 79–82. IEEE (2012)
Nitta, T.: Orthogonality of decision boundaries in complex-valued neural networks. Neural Comput. 16(1), 73–97 (2004)
Philipp, G., Song, D., Carbonell, J.G.: The exploding gradient problem demystified-definition, prevalence, impact, origin, tradeoffs, and solutions. arXiv preprint arXiv:1712.05577 (2017)
Rabiner, L.R., Gold, B.: Theory and Application of Digital Signal Processing. Prentice-Hall, Englewood Cliffs (1975)
Rosenblatt, F.: The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory (1957)
Trabelsi, C., et al.: Deep complex networks. arXiv preprint arXiv:1705.09792 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guimerà Cuevas, F., Phan, T., Schmid, H. (2023). Adaptive Bi-nonlinear Neural Networks Based on Complex Numbers with Weights Constrained Along the Unit Circle. In: Kashima, H., Ide, T., Peng, WC. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2023. Lecture Notes in Computer Science(), vol 13935. Springer, Cham. https://doi.org/10.1007/978-3-031-33374-3_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-33374-3_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-33373-6
Online ISBN: 978-3-031-33374-3
eBook Packages: Computer ScienceComputer Science (R0)