Abstract
Multilayer Perceptrons, Recurrent neural networks, Convolutional networks, and others types of neural networks are widespread nowadays. Neural Networks have hyperparameters like number of hidden layers, number of units for each hidden layer, learning rate, and activation function. Bayesian Optimization is one of the methods used for tuning hyperparameters. Usually this technique treats values of neurons in network as stochastic Gaussian processes. This article reports experimental results on multivariate normality test and proves that the neuron vectors are considerably far from Gaussian distribution.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
REFERENCES
A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in Proc.2014IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2014) (Columbus, OH, USA, 23–28 June 2014), pp. 1725–1732.
G. I. Diaz, A. Fokoue-Nkoutche, G. Nannicini, and H. Samulowitz, “An effective algorithm for hyperparameter optimization of neural networks,” IBM J. Res. Dev. 61 (4/5), 9.1 – 9.11 (2017).
T. Domhan, J. T. Springenberg, and F. Hutter, “Speeding up automatic hyperparameter optimization of Deep Neural Networks by extrapolation of learning curves,” in Proc. 24th Int. Joint Conf. on Artificial Intelligence (IJCAI2015) (Buenos Aires, Argentina, 25–31 July 2015), pp. 3460–3460.
J. Bergstra and B. Yoshua, “Random search for hyperparameter optimization,” J. Mach. Learn. Res. 13 (2), 281–305 (2012).
J. Bergstra, D. Yamins, and D. D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in Proc. 30th Int. Conf. on Machine Learning (ICLM’13) (Atlanta, Georgia, USA. 16–21 June 2013), JMLR: W&CP 28, I-115–I-123 (2013).
D. Orive, G. Sorrosal, C. E. Borges, C. Martin, and A. Alonso-Vicario, “Evolutionary algorithms for hyperparameter tuning on neural networks models,” in Proc. 26th European Modeling and Simulation Symposium (EMSS2014) (Bordeaux, France, 10-12 September 2014), pp. 402–410.
E. Bochinski, T. Senst, and T. Sikora, “Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms,” in Proc.2017IEEE Int. Conf. on Image Processing (ICIP 2017) (Beijing, China, 17-20 September 2017), pp. 3924–3928.
J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” in Advances in Neural Information Processing Systems 25: Proc. Conf. NIPS2012 (Lake Tahoe, Nevada, USA, 3-6 December 2012), pp. 2951–2959.
J. Lampinen and A. Vehtari, “Bayesian approach for neural networks — review and case studies,” Neural Networks 14 (3), 257–274 (2001).
Machine Learning Data Sets, University of California, Irvine. https://archive.ics.uci.edu/ml/datasets.php (Retrieved January 6, 2020).
K.V. Mardia, “Measures of multivariate skewness and kurtosis with applications,” Biometrika, 57 (3), 519–530 (1970).
K.V. Mardia, “Applications of some measures of multivariate skewness and kurtosis for testing normality and robustness studies,” Sankhyā: Indian J. Stat. Ser. B 36 (2), 115–128 (1974).
N. Henze. “Invariant tests for multivariate normality: a critical review,” Stat. Pap. 43 (4), 467–506 (2002).
J. E. Gentle, Computational Statistics (Springer, New York, 2009), pp. 315–316.
L. Baringhaus and N. Henze. “A consistent test for multivariate normality based on the empirical characteristic function,” Metrika 35, 339–348 (1988).
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors declare that they have no conflicts of interest.
Additional information
Youmna Karaki (born in 1983) has graduated from Arts, Sciences, and Technology University In Lebanon in 2005 where she got her Masters degree in Computer Science. She is now a PhD student in the field of Artificial Neural Networks at Belarusian State University of Informatics and Radioelectronics, Minsk.
Y. Karaki has published 2 articles for now. She has more than 15 years of teaching experience at different Lebanese Universities. She is currently working as an Instructor at Arts, Sciences, and Technology University in Lebanon.
Nick Ivanov (born in 1949) has graduated from Belarusian State University in 1972; his specialty is applied mathematics. His fields of interest are network security and artificial neural networks.
N. Ivanov has published 1 monograph and more than 70 papers. He works now as an Associate Professor at Belarusian State University of Informatics and Radioelectronics.
Rights and permissions
About this article
Cite this article
Karaki, Y., Ivanov, N. Hyperparameters of Multilayer Perceptron with Normal Distributed Weights. Pattern Recognit. Image Anal. 30, 170–173 (2020). https://doi.org/10.1134/S1054661820020054
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1054661820020054