Advertisement

International Journal of Speech Technology

, Volume 22, Issue 4, pp 1039–1049 | Cite as

Evaluation of PNN pattern-layer activation function approximations in different training setups

  • Nikolay T. Dukov
  • Todor D. GanchevEmail author
  • Michael N. Vrahatis
Article
  • 27 Downloads

Abstract

The processing of inputs in the first two layers of the probabilistic neural network (PNN) is highly parallel which makes it quite appropriate for hardware implementations with FPGA. One of the main inconveniences however remains the implementation of the nonlinear activation function of the pattern layer neurons. In the present study, we investigate the applicability of three approximations of the exponential activation function with look-up tables of different precision and the effect this has on the training process and the classification accuracy. Furthermore, seeking for a highly-parallel hardware-friendly algorithm for the automated adjustment of the spread factor \(\sigma _i\), we investigated the performance of fifteen PNN training setups, which are based on the differential evolution (DE) or unified particle swarm optimization (UPSO) methods. The experimental evaluation was performed following a common experimental protocol, which makes use of the Parkinson Speech Dataset, as this research aims to support the development of portable medical devices that are capable to detect episodes with exacerbation in patients with Parkinson’s disease. The performance of the most successful setups is discussed in terms of error rates and from the perspective of the resources required for an FPGA-based implementation.

Keywords

Probabilistic neural network Differential evolution Particle swarm optimization Parkinson speech dataset Hardware-friendly algorithm 

Abbreviations

CPU

Central processing unit

DE

Differential evolution

EA

Evolutionary algorithm

FPGA

Field-programmable gate array

GA

Genetic algorithm

GPU

Graphics processing unit

LUT

Look-up table

PDF

Probability density function

PNN

Probabilistic neural network

PSO

Particle swarm optimization

UPSO

Unified particle swarm optimization

Notes

Acknowledgements

The authors T.D.G. and N.T.D. acknowledge with thanks the support received through the research projects PD5 “Study of biologically substantiated architectures of artificial neural networks for the identification of heart diseases and neurological disorders”, SNP2 “Technological support for improving quality of life of people with the Alzheimer disease”, and the NP4 “Capacity building for object-oriented FPGA design in support of knowledge-based economy” financed by the Technical University of Varna and the Bulgarian National Science Fund. In addition, N.T.D. would like to acknowledge the financial support of the National Science Program “Young Scientists and Postdoctoral Students” financed by the state budget and the Erasmus Mundus Programme of the European Commission. Also, to thank Assoc. Prof. Dimitar Kovachev for the fruitful discussion on certain aspects of FPGA design.

References

  1. Acharya, U. R., Mookiah, M. R. K., Sree, S. V., Yanti, R., Martis, R., Saba, L., et al. (2013). Evolutionary algorithm-based classifier parameter tuning for automatic ovarian cancer tissue characterization and classification. In L. Saba, U. Acharya, S. Guerriero, & J. Suri (Eds.), Ovarian neoplasm imaging (pp. 425–440). Boston: Springer.  https://doi.org/10.1007/978-1-4614-8633-6_27 CrossRefGoogle Scholar
  2. Akhmetov, Y., & James, A.P. (2019). Probabilistic neural network with memristive crossbar circuits. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1–5).  https://doi.org/10.1109/ISCAS.2019.8702153
  3. Alshurafa, N., Eastwood, J., Pourhomayoun, M., Liu, J.J., & Sarrafzadeh, M. (2014). Remote health monitoring: Predicting outcome success based on contextual features for cardiovascular disease. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 1777–1781).  https://doi.org/10.1109/EMBC.2014.6943953
  4. Behera, M., Fowler, E. E., Owonikoko, T. K., Land, W. H., Mayfield, W., Chen, Z., et al. (2011). Statistical learning methods as a preprocessing step for survival analysis: Evaluation of concept using lung cancer data. BioMedical Engineering OnLine, 10(1), 97.  https://doi.org/10.1186/1475-925X-10-97.CrossRefGoogle Scholar
  5. Birk, M., Balzer, M., Ruiter, N., & Becker, J. (2012). Comparison of processing performance and architectural efficiency metrics for FPGAs and GPUs in 3D ultrasound computer tomography. In 2012 International Conference on Reconfigurable Computing and FPGAs (pp. 1–7).  https://doi.org/10.1109/ReConFig.2012.6416735
  6. Boulos, M. N. K., Brewer, A. C., Karimkhani, C., Buller, D. B., & Dellavalle, R. P. (2014). Mobile medical and health apps: State of the art, concerns, regulatory control and certification. Online Journal of Public Health Informatics, 5(3), 229.  https://doi.org/10.5210/ojphi.v5i3.4814.CrossRefGoogle Scholar
  7. Brandenburg, C., Worrall, L., Rodriguez, A., & Copland, D. (2013). Mobile computing technology and aphasia: An integrated review of accessibility and potential uses. Aphasiology, 27(4), 444–461.  https://doi.org/10.1080/02687038.2013.772293.CrossRefGoogle Scholar
  8. Chen, C. H., Chang, H. W., & Kuo, C. M. (2019). VLSI implementation of anisotropic probabilistic neural network for real-time image scaling. Journal of Real-Time Image Processing, 16(1), 71–80.  https://doi.org/10.1007/s11554-018-0770-3.MathSciNetCrossRefGoogle Scholar
  9. Ciarelli, P., Krohling, R., & Oliveira, E. (2009). Particle swarm optimization applied to parameters learning of probabilistic neural networks for classification of economic activities. In Particle Swarm Optimization, InTech, chap 19,  https://doi.org/10.5772/6756 Google Scholar
  10. Clerc, M., & Kennedy, J. (2002). The particle swarm—Explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation, 6(1), 58–73.  https://doi.org/10.1109/4235.985692.CrossRefGoogle Scholar
  11. Derrac, J., Garcia, S., Molina, D., & Herrera, F. (2011). A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation, 1(1), 3–18.  https://doi.org/10.1016/j.swevo.2011.02.002.CrossRefGoogle Scholar
  12. Ford, W., Xiang, K., Land, W., Congdon, R., Li, Y., & Sadik, O. (2013). A multi-class probabilistic neural network for pathogen classification. Procedia Computer Science, 20, 348–353.  https://doi.org/10.1016/j.procs.2013.09.284.CrossRefGoogle Scholar
  13. Ganchev, T. (2009). Enhanced training for the locally recurrent probabilistic neural networks. International Journal of Artificial Intelligence Tools, 18(6), 853–881.  https://doi.org/10.1142/S0218213009000433.CrossRefGoogle Scholar
  14. Georgiou, V. L., Pavlidis, N. G., Parsopoulos, K. E., Alevizos, P. D., & Vrahatis, M. N. (2006). New self-adaptive probabilistic neural networks in bioinformatics and medical tasks. International Journal on Artificial Intelligence Tools, 15(3), 371–396.  https://doi.org/10.1142/S0218213006002722.CrossRefGoogle Scholar
  15. Georgiou, V. L., Alevizos, P. D., & Vrahatis, M. N. (2008). Novel approaches to probabilistic neural networks through bagging and evolutionary estimating of prior probabilities. Neural Processing Letters, 27(2), 153–162.  https://doi.org/10.1007/s11063-007-9066-5.CrossRefzbMATHGoogle Scholar
  16. Hsieh, S., & Chen, C. (2009). Adaptive image interpolation using probabilistic neural network. Expert Systems with Applications, 36(3), 6025–6029.  https://doi.org/10.1016/j.eswa.2008.06.124.CrossRefGoogle Scholar
  17. Huang, D., & Du, J. (2008). A constructive hybrid structure optimization methodology for radial basis probabilistic neural networks. IEEE Transactions on Neural Networks, 19(12), 2099–2115.  https://doi.org/10.1109/TNN.2008.2004370.CrossRefGoogle Scholar
  18. Hussain, H.M., Benkrid, K., Erdogan, A.T., & Seker, H. (2011). Highly parameterized k-means clustering on FPGAs: Comparative results with GPPs and GPUs. In Proceedings of the 2011 International Conference on Reconfigurable Computing and FPGAs, RECONFIG ’11 pp. 475–480.  https://doi.org/10.1109/ReConFig.2011.49
  19. Zhang, Jianhai, Chen, Ming, Hu, Sanqing, Cao, Yu, & Kozma, R. (2016). Pnn for eeg-based emotion recognition. In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 002319–002323).  https://doi.org/10.1109/SMC.2016.7844584
  20. Kennedy, J., & Eberhart, R. (1995). Particle swarm optimization. Proceedings of ICNN’95—International Conference on Neural Networks (pp. 1942–1948).  https://doi.org/10.1109/ICNN.1995.488968
  21. Krestinskaya, O., & James, A.P. (2018). Approximate probabilistic neural networks with gated threshold logic. In 2018 IEEE 18th International Conference on Nanotechnology (IEEE-NANO) (pp. 1–4).  https://doi.org/10.1109/NANO.2018.8626302
  22. Kusy, M., & Zajdel, R. (2015). Application of reinforcement learning algorithms for the adaptive computation of the smoothing parameter for probabilistic neural network. IEEE Transactions on Neural Networks and Learning Systems, 26(9), 2163–2175.  https://doi.org/10.1109/TNNLS.2014.2376703.MathSciNetCrossRefGoogle Scholar
  23. Lavanyadevi, R., Machakowsalya, M., Nivethitha, J., & Kumar, A.N. (2017). Brain tumor classification and segmentation in mri images using pnn. In 2017 IEEE International Conference on Electrical, Instrumentation and Communication Engineering (ICEICE) (pp. 1–6).  https://doi.org/10.1109/ICEICE.2017.8191888
  24. Li, L., & Ma, G. (2008). Optimizing the performance of probabilistic neural networks using PSO in the task of traffic sign recognition. In Advanced Intelligent Computing Theories and Applications. With Aspects of Artificial Intelligence (pp. 90–98)  https://doi.org/10.1007/978-3-540-85984-0_12 CrossRefGoogle Scholar
  25. Mao, K. Z., Tan, K., & Ser, W. (2000). Probabilistic neural-network structure determination for pattern classification. IEEE Transactions on Neural Networks, 11(4), 1009–1016.  https://doi.org/10.1109/72.857781.CrossRefGoogle Scholar
  26. Mathew, N.A., Vivek, R.S., & Anurenjan, P.R. (2018). Early diagnosis of alzheimer’s disease from mri images using pnn. In 2018 International CET Conference on Control, Communication, and Computing (IC4) (pp. 161–164)  https://doi.org/10.1109/CETIC4.2018.8530910
  27. Mittal, S., & Vetter, J. S. (2014). A survey of methods for analyzing and improving GPU energy efficiency. ACM Computing Surveys (CSUR), 47(2), 19:1–19:23.  https://doi.org/10.1145/2636342.CrossRefGoogle Scholar
  28. Moser, L.E., & Melliar-Smith, P.M. (2015). Personal health monitoring using a smartphone. In 2015 IEEE International Conference on Mobile Services (pp. 344–351)  https://doi.org/10.1109/MobServ.2015.54
  29. Palo, H. K., Chandra, M., & Mohanty, M. N. (2018). Recognition of human speech emotion using variants of Mel-frequency cepstral coefficients (pp. 491–498). Singapore: Springer.  https://doi.org/10.1007/978-981-10-4762-6_47.CrossRefGoogle Scholar
  30. Parisi, L., RaviChandran, N., & Manaog, M. L. (2018). Feature-driven machine learning to improve early diagnosis of Parkinson’s disease. Expert Systems with Applications, 110, 182–190.  https://doi.org/10.1016/j.eswa.2018.06.003.CrossRefGoogle Scholar
  31. Parsopoulos, K., & Vrahatis, M. (2002). Recent approaches to global optimization problems through particle swarm optimization. Natural Computing, 1(2), 235–306.  https://doi.org/10.1023/A:1016568309421.MathSciNetCrossRefzbMATHGoogle Scholar
  32. Parsopoulos, K., & Vrahatis, M. (2005). Unified particle swarm optimization for tackling operations research problems. In Swarm intelligence symposium, 2005 SIS 2005 Proceedings 2005 IEEE (pp. 53–59)  https://doi.org/10.1109/SIS.2005.1501602
  33. Parsopoulos, K., & Vrahatis, M. (2007). Parameter selection and adaptation in unified particle swarm optimization. Mathematical and Computer Modelling, 46(1–2), 198–213.  https://doi.org/10.1016/j.mcm.2006.12.019.MathSciNetCrossRefzbMATHGoogle Scholar
  34. Parsopoulos, K., & Vrahatis, M. (2010). Particle swarm optimization and intelligence: Advances and applications. Hershey, PA: Information Science Publishing (IGI Global).  https://doi.org/10.4018/978-1-61520-666-7.CrossRefGoogle Scholar
  35. Plagianakos, V. P., & Vrahatis, M. N. (2002). Parallel evolutionary training algorithms for “hardware-friendly” neural networks. Natural Computing, 1(2), 307–322.  https://doi.org/10.1023/A:1016545907026.MathSciNetCrossRefzbMATHGoogle Scholar
  36. Sakar, B. E., Isenkul, M., Sakar, C., Sertbas, A., Gurgen, F., Delil, S., et al. (2013). Collection and analysis of a Parkinson speech dataset with multiple types of sound recordings. IEEE Journal of Biomedical and Health Informatics, 17(4), 828–834.  https://doi.org/10.1109/JBHI.2013.2245674.CrossRefGoogle Scholar
  37. Samanta, B., Al-Balushi, K. R., & Al-Araimi, S. A. (2006). Artificial neural networks and genetic algorithm for bearing fault detection. Soft Computing, 10(3), 264–271.  https://doi.org/10.1007/s00500-005-0481-0.CrossRefGoogle Scholar
  38. Specht, D. (1990). Probabilistic neural networks. Neural Networks, 3, 109–118.  https://doi.org/10.1016/0893-6080(90)90049-Q.CrossRefGoogle Scholar
  39. Storn, R., & Price, K. (1997). Differential evolution—A simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 11(4), 341–359.  https://doi.org/10.1023/A:1008202821328.MathSciNetCrossRefzbMATHGoogle Scholar
  40. Trelea, I. C. (2003). The particle swarm optimization algorithm: Convergence analysis and parameter selection. Information Processing Letters, 85(6), 317–325.  https://doi.org/10.1016/S0020-0190(02)00447-7.MathSciNetCrossRefzbMATHGoogle Scholar
  41. Ullah, I., Hussain, M., ul Haq, Qazi E., & Aboalsamh, H. (2018). An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Systems with Applications, 107, 61–71.  https://doi.org/10.1016/j.eswa.2018.04.021.CrossRefGoogle Scholar
  42. Van Essen, B.C., Macaraeg, C.C., Prenger, R., & Gokhale, M. (2012). Accelerating a random forest classifier: Multi-core, GP-GPU, or FPGA? In International Symposium on Field-Programmable Custom Computing Machines (FCCM’12). IEEE pp. (232–239).  https://doi.org/10.1109/FCCM.2012.47
  43. Vipin, K., Akhmetov, Y., Myrzakhme, S., & James, A.P. (2018). Fapnn: An fpga based approximate probabilistic neural network library. In 2018 International Conference on Computing and Network Communications (CoCoNet) (pp. 64–68).  https://doi.org/10.1109/CoCoNet.2018.8476889
  44. Wang, D., Hao, Y., Zhu, X., Zhao, T., Wang, Y., Chen, Y., Chen, W., & Zheng, X. (2011). FPGA implementation of hardware processing modules as coprocessors in brain-machine interfaces. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4613–4616).  https://doi.org/10.1109/IEMBS.2011.6091142
  45. Chtioui, Y., Marsh, R. A., & Panigrahi, S. (1998). Conjugate gradient and approximate Newton methods for an optimal probablilistic neural network for food color classification. Optical Engineering, 37(11), 3015–3024.  https://doi.org/10.1117/1.601972.CrossRefGoogle Scholar
  46. Zhou, F., Liu, J., Yu, Y., Tian, X., Liu, H., Hao, Y., et al. (2010). Field-programmable gate array implementation of a probabilistic neural network for motor cortical decoding in rats. Journal of Neuroscience Methods, 185(2), 299–306.  https://doi.org/10.1016/j.jneumeth.2009.10.001.CrossRefGoogle Scholar
  47. Zhou, J., Zhong, T., & He, X. (2017). Auxiliary diagnosis of breast tumor based on pnn classifier optimized by pca and pso algorithm. In 2017 9th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 222–227).  https://doi.org/10.1109/IHMSC.2017.164

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Applied Signal Processing Laboratory (ASPL), Faculty of Computing and AutomationTechnical University of VarnaVarnaBulgaria
  2. 2.Computational Intelligence Laboratory (CI Lab), Department of MathematicsUniversity of PatrasPatrasGreece

Personalised recommendations