International Journal of Speech Technology

, Volume 22, Issue 4, pp 1039–1049 | Cite as

Evaluation of PNN pattern-layer activation function approximations in different training setups

  • Nikolay T. Dukov
  • Todor D. GanchevEmail author
  • Michael N. Vrahatis


The processing of inputs in the first two layers of the probabilistic neural network (PNN) is highly parallel which makes it quite appropriate for hardware implementations with FPGA. One of the main inconveniences however remains the implementation of the nonlinear activation function of the pattern layer neurons. In the present study, we investigate the applicability of three approximations of the exponential activation function with look-up tables of different precision and the effect this has on the training process and the classification accuracy. Furthermore, seeking for a highly-parallel hardware-friendly algorithm for the automated adjustment of the spread factor \(\sigma _i\), we investigated the performance of fifteen PNN training setups, which are based on the differential evolution (DE) or unified particle swarm optimization (UPSO) methods. The experimental evaluation was performed following a common experimental protocol, which makes use of the Parkinson Speech Dataset, as this research aims to support the development of portable medical devices that are capable to detect episodes with exacerbation in patients with Parkinson’s disease. The performance of the most successful setups is discussed in terms of error rates and from the perspective of the resources required for an FPGA-based implementation.


Probabilistic neural network Differential evolution Particle swarm optimization Parkinson speech dataset Hardware-friendly algorithm 



Central processing unit


Differential evolution


Evolutionary algorithm


Field-programmable gate array


Genetic algorithm


Graphics processing unit


Look-up table


Probability density function


Probabilistic neural network


Particle swarm optimization


Unified particle swarm optimization



The authors T.D.G. and N.T.D. acknowledge with thanks the support received through the research projects PD5 “Study of biologically substantiated architectures of artificial neural networks for the identification of heart diseases and neurological disorders”, SNP2 “Technological support for improving quality of life of people with the Alzheimer disease”, and the NP4 “Capacity building for object-oriented FPGA design in support of knowledge-based economy” financed by the Technical University of Varna and the Bulgarian National Science Fund. In addition, N.T.D. would like to acknowledge the financial support of the National Science Program “Young Scientists and Postdoctoral Students” financed by the state budget and the Erasmus Mundus Programme of the European Commission. Also, to thank Assoc. Prof. Dimitar Kovachev for the fruitful discussion on certain aspects of FPGA design.


  1. Acharya, U. R., Mookiah, M. R. K., Sree, S. V., Yanti, R., Martis, R., Saba, L., et al. (2013). Evolutionary algorithm-based classifier parameter tuning for automatic ovarian cancer tissue characterization and classification. In L. Saba, U. Acharya, S. Guerriero, & J. Suri (Eds.), Ovarian neoplasm imaging (pp. 425–440). Boston: Springer. CrossRefGoogle Scholar
  2. Akhmetov, Y., & James, A.P. (2019). Probabilistic neural network with memristive crossbar circuits. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1–5).
  3. Alshurafa, N., Eastwood, J., Pourhomayoun, M., Liu, J.J., & Sarrafzadeh, M. (2014). Remote health monitoring: Predicting outcome success based on contextual features for cardiovascular disease. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 1777–1781).
  4. Behera, M., Fowler, E. E., Owonikoko, T. K., Land, W. H., Mayfield, W., Chen, Z., et al. (2011). Statistical learning methods as a preprocessing step for survival analysis: Evaluation of concept using lung cancer data. BioMedical Engineering OnLine, 10(1), 97. Scholar
  5. Birk, M., Balzer, M., Ruiter, N., & Becker, J. (2012). Comparison of processing performance and architectural efficiency metrics for FPGAs and GPUs in 3D ultrasound computer tomography. In 2012 International Conference on Reconfigurable Computing and FPGAs (pp. 1–7).
  6. Boulos, M. N. K., Brewer, A. C., Karimkhani, C., Buller, D. B., & Dellavalle, R. P. (2014). Mobile medical and health apps: State of the art, concerns, regulatory control and certification. Online Journal of Public Health Informatics, 5(3), 229. Scholar
  7. Brandenburg, C., Worrall, L., Rodriguez, A., & Copland, D. (2013). Mobile computing technology and aphasia: An integrated review of accessibility and potential uses. Aphasiology, 27(4), 444–461. Scholar
  8. Chen, C. H., Chang, H. W., & Kuo, C. M. (2019). VLSI implementation of anisotropic probabilistic neural network for real-time image scaling. Journal of Real-Time Image Processing, 16(1), 71–80. Scholar
  9. Ciarelli, P., Krohling, R., & Oliveira, E. (2009). Particle swarm optimization applied to parameters learning of probabilistic neural networks for classification of economic activities. In Particle Swarm Optimization, InTech, chap 19, Google Scholar
  10. Clerc, M., & Kennedy, J. (2002). The particle swarm—Explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation, 6(1), 58–73. Scholar
  11. Derrac, J., Garcia, S., Molina, D., & Herrera, F. (2011). A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation, 1(1), 3–18. Scholar
  12. Ford, W., Xiang, K., Land, W., Congdon, R., Li, Y., & Sadik, O. (2013). A multi-class probabilistic neural network for pathogen classification. Procedia Computer Science, 20, 348–353. Scholar
  13. Ganchev, T. (2009). Enhanced training for the locally recurrent probabilistic neural networks. International Journal of Artificial Intelligence Tools, 18(6), 853–881. Scholar
  14. Georgiou, V. L., Pavlidis, N. G., Parsopoulos, K. E., Alevizos, P. D., & Vrahatis, M. N. (2006). New self-adaptive probabilistic neural networks in bioinformatics and medical tasks. International Journal on Artificial Intelligence Tools, 15(3), 371–396. Scholar
  15. Georgiou, V. L., Alevizos, P. D., & Vrahatis, M. N. (2008). Novel approaches to probabilistic neural networks through bagging and evolutionary estimating of prior probabilities. Neural Processing Letters, 27(2), 153–162. Scholar
  16. Hsieh, S., & Chen, C. (2009). Adaptive image interpolation using probabilistic neural network. Expert Systems with Applications, 36(3), 6025–6029. Scholar
  17. Huang, D., & Du, J. (2008). A constructive hybrid structure optimization methodology for radial basis probabilistic neural networks. IEEE Transactions on Neural Networks, 19(12), 2099–2115. Scholar
  18. Hussain, H.M., Benkrid, K., Erdogan, A.T., & Seker, H. (2011). Highly parameterized k-means clustering on FPGAs: Comparative results with GPPs and GPUs. In Proceedings of the 2011 International Conference on Reconfigurable Computing and FPGAs, RECONFIG ’11 pp. 475–480.
  19. Zhang, Jianhai, Chen, Ming, Hu, Sanqing, Cao, Yu, & Kozma, R. (2016). Pnn for eeg-based emotion recognition. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 002319–002323).
  20. Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN’95—International Conference on Neural Networks (pp. 1942–1948).
  21. Krestinskaya, O., & James, A.P. (2018). Approximate probabilistic neural networks with gated threshold logic. In 2018 IEEE 18th International Conference on Nanotechnology (IEEE-NANO) (pp. 1–4).
  22. Kusy, M., & Zajdel, R. (2015). Application of reinforcement learning algorithms for the adaptive computation of the smoothing parameter for probabilistic neural network. IEEE Transactions on Neural Networks and Learning Systems, 26(9), 2163–2175. Scholar
  23. Lavanyadevi, R., Machakowsalya, M., Nivethitha, J., & Kumar, A.N. (2017). Brain tumor classification and segmentation in mri images using pnn. In 2017 IEEE International Conference on Electrical, Instrumentation and Communication Engineering (ICEICE) (pp. 1–6).
  24. Li, L., & Ma, G. (2008). Optimizing the performance of probabilistic neural networks using PSO in the task of traffic sign recognition. In Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelligence (pp. 90–98) CrossRefGoogle Scholar
  25. Mao, K. Z., Tan, K., & Ser, W. (2000). Probabilistic neural-network structure determination for pattern classification. IEEE Transactions on Neural Networks, 11(4), 1009–1016. Scholar
  26. Mathew, N.A., Vivek, R.S., & Anurenjan, P.R. (2018). Early diagnosis of alzheimer’s disease from mri images using pnn. In 2018 International CET Conference on Control, Communication, and Computing (IC4) (pp. 161–164)
  27. Mittal, S., & Vetter, J. S. (2014). A survey of methods for analyzing and improving GPU energy efficiency. ACM Computing Surveys (CSUR), 47(2), 19:1–19:23. Scholar
  28. Moser, L.E., & Melliar-Smith, P.M. (2015). Personal health monitoring using a smartphone. In 2015 IEEE International Conference on Mobile Services (pp. 344–351)
  29. Palo, H. K., Chandra, M., & Mohanty, M. N. (2018). Recognition of human speech emotion using variants of Mel-frequency cepstral coefficients (pp. 491–498). Singapore: Springer. Scholar
  30. Parisi, L., RaviChandran, N., & Manaog, M. L. (2018). Feature-driven machine learning to improve early diagnosis of Parkinson’s disease. Expert Systems with Applications, 110, 182–190. Scholar
  31. Parsopoulos, K., & Vrahatis, M. (2002). Recent approaches to global optimization problems through particle swarm optimization. Natural Computing, 1(2), 235–306. Scholar
  32. Parsopoulos, K., & Vrahatis, M. (2005). Unified particle swarm optimization for tackling operations research problems. In Swarm intelligence symposium, 2005 SIS 2005 Proceedings 2005 IEEE (pp. 53–59)
  33. Parsopoulos, K., & Vrahatis, M. (2007). Parameter selection and adaptation in unified particle swarm optimization. Mathematical and Computer Modelling, 46(1–2), 198–213. Scholar
  34. Parsopoulos, K., & Vrahatis, M. (2010). Particle swarm optimization and intelligence: Advances and applications. Hershey, PA: Information Science Publishing (IGI Global). Scholar
  35. Plagianakos, V. P., & Vrahatis, M. N. (2002). Parallel evolutionary training algorithms for “hardware-friendly” neural networks. Natural Computing, 1(2), 307–322. Scholar
  36. Sakar, B. E., Isenkul, M., Sakar, C., Sertbas, A., Gurgen, F., Delil, S., et al. (2013). Collection and analysis of a Parkinson speech dataset with multiple types of sound recordings. IEEE Journal of Biomedical and Health Informatics, 17(4), 828–834. Scholar
  37. Samanta, B., Al-Balushi, K. R., & Al-Araimi, S. A. (2006). Artificial neural networks and genetic algorithm for bearing fault detection. Soft Computing, 10(3), 264–271. Scholar
  38. Specht, D. (1990). Probabilistic neural networks. Neural Networks, 3, 109–118. Scholar
  39. Storn, R., & Price, K. (1997). Differential evolution—A simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 11(4), 341–359. Scholar
  40. Trelea, I. C. (2003). The particle swarm optimization algorithm: Convergence analysis and parameter selection. Information Processing Letters, 85(6), 317–325. Scholar
  41. Ullah, I., Hussain, M., ul Haq, Qazi E., & Aboalsamh, H. (2018). An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Systems with Applications, 107, 61–71. Scholar
  42. Van Essen, B.C., Macaraeg, C.C., Prenger, R., & Gokhale, M. (2012). Accelerating a random forest classifier: Multi-core, GP-GPU, or FPGA? In International Symposium on Field-Programmable Custom Computing Machines (FCCM’12). IEEE pp. (232–239).
  43. Vipin, K., Akhmetov, Y., Myrzakhme, S., & James, A.P. (2018). Fapnn: An fpga based approximate probabilistic neural network library. In 2018 International Conference on Computing and Network Communications (CoCoNet) (pp. 64–68).
  44. Wang, D., Hao, Y., Zhu, X., Zhao, T., Wang, Y., Chen, Y., Chen, W., & Zheng, X. (2011). FPGA implementation of hardware processing modules as coprocessors in brain-machine interfaces. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4613–4616).
  45. Chtioui, Y., Marsh, R. A., & Panigrahi, S. (1998). Conjugate gradient and approximate Newton methods for an optimal probablilistic neural network for food color classification. Optical Engineering, 37(11), 3015–3024. Scholar
  46. Zhou, F., Liu, J., Yu, Y., Tian, X., Liu, H., Hao, Y., et al. (2010). Field-programmable gate array implementation of a probabilistic neural network for motor cortical decoding in rats. Journal of Neuroscience Methods, 185(2), 299–306. Scholar
  47. Zhou, J., Zhong, T., & He, X. (2017). Auxiliary diagnosis of breast tumor based on pnn classifier optimized by pca and pso algorithm. In 2017 9th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 222–227).

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Applied Signal Processing Laboratory (ASPL), Faculty of Computing and AutomationTechnical University of VarnaVarnaBulgaria
  2. 2.Computational Intelligence Laboratory (CI Lab), Department of MathematicsUniversity of PatrasPatrasGreece

Personalised recommendations