Non-uniform Initialization of Inputs Groupings in Contextual Neural Networks
Contextual neural networks which are using neurons with conditional aggregation functions were found to be efficient and useful generalizations of classical multilayer perceptrons. They allow to generate neural classification models with good generalization and low activity of connections between neurons in hidden layers. The key factor to build such solutions is achieving self-consistency between continuous values of weights of neurons’ connections and their mutually related non-continuous aggregation priorities. This allows to optimize neuron inputs aggregation priorities by simultaneous gradient-based optimization of connections’ weights with generalized BP algorithm. But such method additionally needs initial setting of connections groupings (scan-paths) to define priorities of signals during first ω epochs of training. In earlier studies all connections were initially assigned to a single group to give neurons access to all input signals at the beginning of training. We found out that such uniform solution not always is the best one. Thus within this text we compare efficiency of training of contextual neural networks with uniform and non-uniform, random initialization of connections groupings. On this basis we also discuss the properties of analyzed training algorithm which are related to characteristics of used scan-paths initialization methods.
KeywordsClassification Self-consistency Scan-paths initialization Aggregation functions
- 1.Huk, M.: Learning distributed selective attention strategies with the Sigma-if neural network. In: Akbar, M., Hussain, D. (eds.) Advances in Computer Science and IT, pp. 209–232. InTech, Vukovar (2009)Google Scholar
- 4.Szczepanik, M., Jóźwiak, I.: Data management for fingerprint recognition algorithm based on characteristic points’ groups. In: Pechenizkiy, M., Wojciechowski, M. (eds.) New Trends in Databases and Information Systems. AISC, vol. 185, pp. 425–432. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-32518-2_40Google Scholar
- 5.Huk, M.: Measuring the effectiveness of hidden context usage by machine learning methods under conditions of increased entropy of noise. In: 2017 3rd IEEE International Conference on Cybernetics (CYBCONF), pp. 1–6. IEEE (2017)Google Scholar
- 8.UCI Machine Learning Repository. http://archive.ics.uci.edu/ml