Spatial and Temporal Feature Extraction Using a Restricted Boltzmann Machine Model

  • Jefferson Hernandez
  • Andres G. AbadEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 833)


A restricted Boltzmann machine (RBM) is a generative neural-network model with many applications, such as, collaborative filtering, acoustic modeling, and topic modeling. An RBM lacks the capacity to retain memory, making it inappropriate for dynamic data modeling as in time-series or video analysis. In this work we address this issue by proposing the p-RBM model: a generalization of the regular RBM model capable of retaining memory of p past states. We further show how to train the p-RBM model using contrastive divergence and test our model on the problem of recognizing human actions from video data using unsupervised feature extraction. Obtained results show that the p-RBM offers promising capabilities for feature-learning in classification applications.


Restricted Boltzmann machines Neural networks Sequential data Video Human action recognition Unsupervised feature extraction 


  1. 1.
    Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep learning for human action recognition. In: Salah, A.A., Lepri, B. (eds.) HBU 2011. LNCS, vol. 7065, pp. 29–39. Springer, Heidelberg (2011). Scholar
  2. 2.
    Boulanger-Lewandowski, N., Bengio, Y., Vincent, P.: Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription (2012)Google Scholar
  3. 3.
    Chen, H.S., Chen, H.T., Chen, Y.W., Lee, S.Y.: Human action recognition using star skeleton. In: Proceedings of the 4th ACM International Workshop on Video Surveillance and Sensor Networks, VSSN 2006, pp. 171–178. ACM, New York (2006).
  4. 4.
    Dahl, G., Ranzato, M.A., Mohamed, A.R., Hinton, G.E.: Phone recognition with the mean-covariance restricted Boltzmann machine. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) Advances in Neural Information Processing Systems, vol. 23, pp. 469–477. Curran Associates Inc. (2010)Google Scholar
  5. 5.
    Fischer, A., Igel, C.: Training restricted Boltzmann machines: an introduction. Pattern Recognit. 47(1), 25–39 (2014)CrossRefGoogle Scholar
  6. 6.
    Foggia, P., Saggese, A., Strisciuglio, N., Vento, M.: Exploiting the deep learning paradigm for recognizing human actions. In: 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 93–98. IEEE (2014)Google Scholar
  7. 7.
    Grushin, A., Monner, D.D., Reggia, J.A., Mishra, A.: Robust human action recognition via long short-term memory. In: 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2013)Google Scholar
  8. 8.
    Hernandez, J., Abad, A.G.: Learning from multivariate discrete sequential data using a restricted Boltzmann machine model. In: 2018 IEEE 1st Colombian Conference on Applications in Computacional Intelligence (ColCACI), pp. 3450–3457. IEEE (2018)Google Scholar
  9. 9.
    Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8), 1771–1800 (2002)CrossRefGoogle Scholar
  10. 10.
    Ji, S., Xu, W., Yang, M., Yu, K.: 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 221–231 (2013)CrossRefGoogle Scholar
  11. 11.
    Laptev, I., Lindeberg, T.: Space-time interest points. In: Proceedings Ninth IEEE International Conference on Computer Vision, vol. 1, pp. 432–439, October 2003.
  12. 12.
    Mnih, V., Larochelle, H., Hinton, G.E.: Conditional restricted Boltzmann machines for structured output prediction. arXiv:1202.3748 [cs, stat], February 2012
  13. 13.
    Nie, S., Ji, Q.: Capturing global and local dynamics for human action recognition. In: 2014 22nd International Conference on Pattern Recognition, pp. 1946–1951, August 2014.
  14. 14.
    Salakhutdinov, R., Mnih, A., Hinton, G.: Restricted Boltzmann machines for collaborative filtering. In: Proceedings of the 24th International Conference on Machine Learning, ICML 2007, pp. 791–798. ACMGoogle Scholar
  15. 15.
    Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local SVM approach. In: 2004 Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 3, pp. 32–36, August 2004.
  16. 16.
    Shu, N., Tang, Q., Liu, H.: A bio-inspired approach modeling spiking neural networks of visual cortex for human action recognition. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 3450–3457. IEEE (2014)Google Scholar
  17. 17.
    Smolensky, P.: Information processing in dynamical systems: foundations of harmony theory (1986)Google Scholar
  18. 18.
    Sun, L., Jia, K., Yeung, D.Y., Shi, B.E.: Human action recognition using factorized spatio-temporal convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4597–4605 (2015)Google Scholar
  19. 19.
    Taylor, G.W., Hinton, G.E.: Factored conditional restricted Boltzmann machines for modeling motion style. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 1025–1032. ACM (2009)Google Scholar
  20. 20.
    Taylor, G.W., Hinton, G.E., Roweis, S.: Modeling human motion using binary latent variables. In: Advances in Neural Information Processing Systems, pp. 1345–1352. MIT Press (2007)Google Scholar
  21. 21.
    Veeriah, V., Zhuang, N., Qi, G.J.: Differential recurrent neural networks for action recognition. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4041–4049. IEEE (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Escuela Superior Politecnica del Litoral (ESPOL)GuayaquilEcuador

Personalised recommendations