Adaptation of RBM Learning for Intel MIC Architecture

  • Tomasz Olas
  • Wojciech K. Mleczko
  • Robert K. Nowicki
  • Roman Wyrzykowski
  • Adam Krzyzak
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9119)

Abstract

In the paper, the parallel realization of the Boltzmann Restricted Machine (RBM) is proposed. The implementation intends to use multicore architectures of modern CPUs and Intel Xeon Phi coprocessor. The learning procedure is based on the matrix description of RBM, where the learning samples are grouped into packages, and represented as matrices. The influence of the package size on convergence of learning, as well as on performance of computation, are studied for various number of threads, using conventional CPU and Intel Phi architecures. Our research confirms a potential usefulness of MIC parallel architecture for implementation of RBM and similar algorithms.

Keywords

Restricted Boltzman Machine Parallel programming Multicore architectures Intel Xeon Phi architecture 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bilski, J., Nowicki, R., Scherer, R., Litwiński, S.: Application of signal processor TMS320c30 to neural networks realisation. In: Proceedings of the Second Conference Neural Networks and Their Applications, Czêstochowa, pp. 53–59 (1996)Google Scholar
  2. 2.
    Bilski, J., Smolag, J.: Parallel architectures for learning the RTRN and Elman dynamic neural networks. IEEE Transactions on Parallel and Distributed Systems PP(99) (2014)Google Scholar
  3. 3.
    Bilski, J., Litwiński, S., Smoląg, J.: Parallel realisation of QR algorithm for neural networks learning. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 158–165. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  4. 4.
    Bilski, J., Smoląg, J., Galushkin, A.I.: The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part I. LNCS (LNAI), vol. 8467, pp. 12–21. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  5. 5.
    Bilski, J., Smoląg, J.: Parallel realisation of the recurrent RTRN neural network learning. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 11–16. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Bilski, J., Smoląg, J.: Parallel realisation of the recurrent elman neural network learning. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010, Part II. LNCS (LNAI), vol. 6114, pp. 19–25. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Bilski, J., Smoląg, J.: Parallel realisation of the recurrent multi layer perceptron learning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 12–20. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  8. 8.
    Bilski, J., Smoląg, J.: Parallel approach to learning of the recurrent Jordan neural network. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013, Part I. LNCS, vol. 7894, pp. 32–40. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  9. 9.
    Chu, J.L., Krzyzak, A.: The recognition of partially occluded objects with support vector machines and convolutional neural networks and deep belief networks. Journal of Artificial Intelligence and Soft Computing Research 4(1), 5–19 (2014)CrossRefGoogle Scholar
  10. 10.
    Intel Corporation: Intel Xeon Phi Coprocessor System Software Developer’s Guide. Technical report, The Intel Corporation (June 2013)Google Scholar
  11. 11.
    Cpałka, K., Rutkowski, L.: Flexible Takagi-Sugeno fuzzy systems. In: Proc. IEEE International Joint Conference on Neural Networks (IJCNN), vol. 3, pp. 1764–1769 (2005)Google Scholar
  12. 12.
    Cpałka, K., Łapa, K., Przybył, A., Zalasiński, M.: A new method for designing neuro-fuzzy systems for nonlinear modelling with interpretability aspects. Neurocomputing 135, 203–217 (2014)CrossRefGoogle Scholar
  13. 13.
    Cpałka, K., Rebrova, O., Nowicki, R., Rutkowski, L.: On design of flexible neuro-fuzzy systems for nonlinear modelling. International Journal of General Systems 42(6), 706–720 (2013)CrossRefMATHGoogle Scholar
  14. 14.
    Dourlens, S., Ramdane-Cherif, A.: Modeling & understanding environment using semantic agents. Journal of Artificial Intelligence and Soft Computing Research 1(4), 301–314 (2011)Google Scholar
  15. 15.
    Fang, J., Varbanescu, A.L., Sips, H.: Benchmarking Intel Xeon Phi to Guide Kernel Design. Delft University of Technology Parallel and Distributed Systems Report Series, No. PDS-2013-005, pp. 1–22 (2013)Google Scholar
  16. 16.
    Gabryel, M., Korytkowski, M., Scherer, R., Rutkowski, L.: Object detection by simple fuzzy classifiers generated by boosting. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013, Part I. LNCS, vol. 7894, pp. 540–547. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  17. 17.
    Galkowski, T., Rutkowski, L.: Nonparametric fitting of multivariate functions. IEEE Transactions on Automatic Control 31(8), 785–787 (1986)CrossRefGoogle Scholar
  18. 18.
    Gałkowski, T.: Kernel estimation of regression functions in the boundary regions. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013, Part II. LNCS, vol. 7895, pp. 158–166. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Galkowski, T., Pawlak, M.: Nonparametric function fitting in the presence of nonstationary noise. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part I. LNCS (LNAI), vol. 8467, pp. 531–538. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  20. 20.
    Gaweda, A.E., Scherer, R.: Fuzzy number-based hierarchical fuzzy system. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 302–307. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  21. 21.
    Hinton, G.: Training products of experts by minimizing contrastive divergence. Neural Computation 14(8), 1771–1800 (2002)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Hinton, G.: A practical guide to training restricted Boltzmann machines. Momentum 9(1), 926 (2010)Google Scholar
  23. 23.
    Hinton, G., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18(7), 1527–1554 (2006)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    http://miclab.pl: MICLAB Pilot laboratory of massively parallel systems. Web Page (2015)
  25. 25.
    http://yann.lecun.com/exdb/mnist/: The mnist database of handwritten digits
  26. 26.
    Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Large-scale video classification with convolutional neural networks. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1725–1732 (June 2014)Google Scholar
  27. 27.
    Korytkowski, M., Rutkowski, L., Scherer, R.: From ensemble of fuzzy classifiers to single fuzzy rule base classifier. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 265–272. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  28. 28.
    Krizhevsky, A.: One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 (2014)Google Scholar
  29. 29.
    Laskowski, L., Laskowska, M.: Functionalization of SBA-15 mesoporous silica by cu-phosphonate units: Probing of synthesis route. Journal of Solid State Chemistry 220, 221–226 (2014)CrossRefGoogle Scholar
  30. 30.
    Laskowski, L., Laskowska, M., Balanda, M., Fitta, M., Kwiatkowska, J., Dzilinski, K., Karczmarska, A.: Mesoporous silica SBA-15 functionalized by nickel-phosphonic units: Raman and magnetic analysis. Microporous and Mesoporous Materials 200, 253–259 (2014)CrossRefGoogle Scholar
  31. 31.
    Laskowski, Ł., Laskowska, M., Jelonkiewicz, J., Boullanger, A.: Spin-glass implementation of a Hopfield neural structure. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part I. LNCS (LNAI), vol. 8467, pp. 89–96. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  32. 32.
    Le Roux, N., Bengio, Y.: Representational power of restricted boltzmann machines and deep belief networks. Neural Computation 20(6), 1631–1649 (2008)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Pabiasz, S., Starczewski, J.: Face reconstruction for 3D systems. In: Rutkowska, D., Cader, A., Przybyszewski, K. (eds.) Selected Topics in Computer Science Applications, pp. 54–63. Academic Publishing House EXIT (2011)Google Scholar
  34. 34.
    Pabiasz, S., Starczewski, J.T.: Meshes vs. depth maps in face recognition systems. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 567–573. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  35. 35.
    Pabiasz, S., Starczewski, J.T., Marvuglia, A.: A new three-dimensional facial landmarks in recognition. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014, Part II. LNCS (LNAI), vol. 8468, pp. 179–186. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  36. 36.
    Pabiasz, S., Starczewski, J.T.: A new approach to determine three-dimensional facial landmarks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013, Part II. LNCS, vol. 7895, pp. 286–296. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  37. 37.
    Patan, K., Patan, M.: Optimal training strategies for locally recurrent neural networks. Journal of Artificial Intelligence and Soft Computing Research 1(2), 103–114 (2011)Google Scholar
  38. 38.
    Reinders, J.: An Overview of Programming for Intel Xeon Processors and Intel Xeon Phi Coprocessors. Technical report, The Intel Corporation (2012)Google Scholar
  39. 39.
    Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review 65(65), 386–408 (1958)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575 (2014)Google Scholar
  41. 41.
    Rutkowski, L., Przybył, A., Cpałka, K., Er, M.J.: Online speed profile generation for industrial machine tool based on neuro-fuzzy approach. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010, Part II. LNCS (LNAI), vol. 6114, pp. 645–650. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  42. 42.
    Saule, E., Kaya, K., Çatalyürek, Ü.V.: Performance Evaluation of Sparse Matrix Multiplication Kernels on Intel Xeon Phi. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2013, Part I. LNCS, vol. 8384, pp. 559–570. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  43. 43.
    Scherer, R., Rutkowski, L.: A fuzzy relational system with linguistic antecedent certainty factors. In: Rutkowski, L., Kacprzyk, J. (eds.) Proceedings of the Sixth International Conference on Neural Network and Soft Computing. Advances in Soft Computing, pp. 563–569. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  44. 44.
    Scherer, R.: Neuro-fuzzy relational systems for nonlinear approximation and prediction. Nonlinear Analysis 71, e1420–e1425 (2009)Google Scholar
  45. 45.
    Smolensky, P.: Information processing in dynamical systems: Foundations of harmony theory. In: Rumelhart, D.E., McLelland, J.L. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Foundations, vol. 1, pp. 194–281. MIT (1986)Google Scholar
  46. 46.
    Staff, C.I., Reinders, J.: Parallel Programming and Optimization with Intel® Xeon PhiTM Coprocessors: Handbook on the Development and Optimization of Parallel Applications for Intel® Xeon Coprocessors and Intel® Xeon PhiTM Coprocessors. Colfax International (2013)Google Scholar
  47. 47.
    Szustak, L., Rojek, K., Gepner, P.: Using Intel Xeon Phi coprocessor to accelerate computations in MPDATA algorithm. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds.) PPAM 2013, Part I. LNCS, vol. 8384, pp. 582–592. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  48. 48.
    Szustak, L., Rojek, K., Olas, T., Kuczynski, L., Halbiniak, K., Gepner, P.: Adaptation of MPDATA heterogeneous stencil computation to Intel Xeon Phi coprocessor. Scientific Programming (in press, 2015)Google Scholar
  49. 49.
    Tambouratzis, T., Chernikova, D., Pázsit, I.: Pulse shape discrimination of neutrons and gamma rays using Kohonen artificial neural networks. Journal of Artificial Intelligence and Soft Computing Research 3(2), 77–88 (2013)CrossRefGoogle Scholar
  50. 50.
    Wyrzykowski, R., Szustak, L., Rojek, K.: Parallelization of 2d MPDATA EULAG algorithm on hybrid architectures with GPU accelerators. Parallel Computing 40(8), 425–447 (2014)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Tomasz Olas
    • 1
  • Wojciech K. Mleczko
    • 2
  • Robert K. Nowicki
    • 2
  • Roman Wyrzykowski
    • 1
  • Adam Krzyzak
    • 3
  1. 1.Institute of Computer and Information SciencesCzestochowa University of TechnologyCzestochowaPoland
  2. 2.Institute of Computational IntelligenceCzestochowa University of TechnologyCzestochowaPoland
  3. 3.Department of Computer Science and Software EngineeringConcordia UniversityMontrealCanada

Personalised recommendations