Action Recognition Using Local Visual Descriptors and Inertial Data

  • Taha AlhershEmail author
  • Samir Brahim Belhaouari
  • Heiner Stuckenschmidt
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11912)


Different body sensors and modalities can be used in human action recognition, either separately or simultaneously. Multi-modal data can be used in recognizing human action. In this work we are using inertial measurement units (IMUs) positioned at left and right hands with first person vision for human action recognition. A novel statistical feature extraction method was proposed based on curvature of the graph of a function and tracking left and right hand positions in space. Local visual descriptors have been used as features for egocentric vision. An intermediate fusion between IMUs and visual sensors has been performed. Despite of using only two IMUs sensors with egocentric vision, our classification result achieved is 99.61% for recognizing nine different actions. Feature extraction step could play a vital step in human action recognition with limited number of sensors, hence, our method might indeed be promising.


Human action recognition IMUs Visual descriptors Feature extraction Classification Sensor fusing 


  1. 1.
    Abebe, G., Cavallaro, A.: Inertial-vision: cross-domain knowledge transfer for wearable sensors. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1392–1400 (2017)Google Scholar
  2. 2.
    Akpinar, S., Alpaslan, F.N.: Video action recognition using an optical flow based representation. In: IPCV, the Steering Committee of the World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), p. 1 (2014)Google Scholar
  3. 3.
    Alhersh, T., Stuckenschmidt, H.: On the combination of IMU and optical flow for action recognition. In: 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE (2019)Google Scholar
  4. 4.
    Alhersh, T., Stuckenschmidt, H.: Unsupervised fine-tuning of optical flow for better motion boundary estimation. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Prague, Czech Republic, 25–27 February 2019. VISAPP, vol. 5. pp. 776–783. SciTePress, Setúbal (2019)., online-Resource
  5. 5.
    Arabacı, M.A., Özkan, F., Surer, E., Jančovič, P., Temizel, A.: Multi-modal egocentric activity recognition using audio-visual features. arXiv preprint arXiv:1807.00612 (2018)
  6. 6.
    Ashry, S., Elbasiony, R., Gomaa, W.: An LSTM-based descriptor for human activities recognition using IMU sensors. In: Proceedings of the 15th International Conference on Informatics in Control, Automation and Robotics, ICINCO, vol. 1, pp. 494–501 (2018)Google Scholar
  7. 7.
    Attal, F., Mohammed, S., Dedabrishvili, M., Chamroukhi, F., Oukhellou, L., Amirat, Y.: Physical human activity recognition using wearable sensors. Sensors 15(12), 31314–31338 (2015)CrossRefGoogle Scholar
  8. 8.
    Betancourt, A., Morerio, P., Regazzoni, C.S., Rauterberg, M.: The evolution of first person vision methods: a survey. IEEE Trans. Circuits Syst. Video Technol. 25(5), 744–760 (2015)CrossRefGoogle Scholar
  9. 9.
    Bevilacqua, A., MacDonald, K., Rangarej, A., Widjaya, V., Caulfield, B., Kechadi, T.: Human activity recognition with convolutional neural networks. In: Brefeld, U., et al. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11053, pp. 541–552. Springer, Cham (2019). Scholar
  10. 10.
    Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2011)CrossRefGoogle Scholar
  11. 11.
    Chen, C., Jafari, R., Kehtarnavaz, N.: A survey of depth and inertial sensor fusion for human action recognition. Multimed. Tools Appl. 76(3), 4405–4425 (2017)CrossRefGoogle Scholar
  12. 12.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  13. 13.
    Coskun, H., Tan, D.J., Conjeti, S., Navab, N., Tombari, F.: Human motion analysis with deep metric learning. arXiv preprint arXiv:1807.11176 (2018)
  14. 14.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection (2005)Google Scholar
  15. 15.
    Dalal, N., Triggs, B., Schmid, C.: Human detection using oriented histograms of flow and appearance. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 428–441. Springer, Heidelberg (2006). Scholar
  16. 16.
    Dasarathy, B.V.: Sensor fusion potential exploitation-innovative architectures and illustrative applications. Proc. IEEE 85(1), 24–38 (1997)CrossRefGoogle Scholar
  17. 17.
    Davila, J.C., Cretu, A.M., Zaremba, M.: Wearable sensor data classification for human activity recognition based on an iterative learning framework. Sensors 17(6), 1287 (2017)CrossRefGoogle Scholar
  18. 18.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)Google Scholar
  19. 19.
    Elmenreich, W.: An introduction to sensor fusion. Vienna University of Technology, Austria 502 (2002)Google Scholar
  20. 20.
    Fortun, D., Bouthemy, P., Kervrann, C.: Optical flow modeling and computation: a survey. Comput. Vis. Image Underst. 134, 1–21 (2015)CrossRefGoogle Scholar
  21. 21.
    Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)CrossRefGoogle Scholar
  22. 22.
    Ijjina, E.P., Chalavadi, K.M.: Human action recognition in RGB-D videos using motion sequence information and deep learning. Pattern Recogn. 72, 504–516 (2017)CrossRefGoogle Scholar
  23. 23.
    Jalloul, N., Porée, F., Viardot, G., LHostis, P., Carrault, G.: Activity recognition using complex network analysis. IEEE J. Biomed. Health Inform. 22(4), 989–1000 (2018)CrossRefGoogle Scholar
  24. 24.
    Kumar, S.S., John, M.: Human activity recognition using optical flow based feature set. In: 2016 IEEE International Carnahan Conference on Security Technology (ICCST), pp. 1–5. IEEE (2016)Google Scholar
  25. 25.
    Lu, Y., Velipasalar, S.: Human activity classification incorporating egocentric video and inertial measurement unit data. In: 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 429–433. IEEE (2018)Google Scholar
  26. 26.
    Moutinho, N.M.B.: Video and image match searching, US Patent App. 15/252,142, 2 March 2017Google Scholar
  27. 27.
    Moya Rueda, F., Grzeszick, R., Fink, G., Feldhorst, S., ten Hompel, M.: Convolutional neural networks for human activity recognition using body-worn sensors. In: Informatics, vol. 5, p. 26. Multidisciplinary Digital Publishing Institute (2018)Google Scholar
  28. 28.
    Nguyen, T.H.C., Nebel, J.C., Florez-Revuelta, F., et al.: Recognition of activities of daily living with egocentric vision: a review. Sensors 16(1), 72 (2016)CrossRefGoogle Scholar
  29. 29.
    Romero, H., Salazar, S., Lozano, R., Benosman, R.: Fusion of optical flow and inertial sensors for four-rotor rotorcraft stabilization. IFAC Proc. Vol. 40(15), 209–214 (2007)CrossRefGoogle Scholar
  30. 30.
    Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)Google Scholar
  31. 31.
    Sevilla-Lara, L., Liao, Y., Guney, F., Jampani, V., Geiger, A., Black, M.J.: On the integration of optical flow and action recognition. arXiv preprint arXiv:1712.08416 (2017)
  32. 32.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  33. 33.
    Singh, S., Arora, C., Jawahar, C.: First person action recognition using deep learned descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2620–2628 (2016)Google Scholar
  34. 34.
    Stein, S., McKenna, S.J.: Recognising complex activities with histograms of relative tracklets. Comput. Vis. Image Underst. 154, 82–93 (2017)CrossRefGoogle Scholar
  35. 35.
    Sudhakaran, S., Escalera, S., Lanz, O.: Hierarchical feature aggregation networks for video action recognition. arXiv preprint arXiv:1905.12462 (2019)
  36. 36.
    Sudhakaran, S., Escalera, S., Lanz, O.: LSTA: long short-term attention for egocentric action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9954–9963 (2019)Google Scholar
  37. 37.
    Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-NET: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943 (2018)Google Scholar
  38. 38.
    Sun, J., Fu, Y., Li, S., He, J., Xu, C., Tan, L.: Sequential human activity recognition based on deep convolutional network and extreme learning machine using wearable sensors. J. Sensors 2018 (2018) Google Scholar
  39. 39.
    Sun, S., Kuang, Z., Sheng, L., Ouyang, W., Zhang, W.: Optical flow guided feature: a fast and robust motion representation for video action recognition. In: CVPR (2018)Google Scholar
  40. 40.
    X-IO Technologies Limited: X-IO technologies limited. UK company (2019).
  41. 41.
    De la Torre, F., et al.: Guide to the Carnegie Mellon university multimodal activity (CMU-MMAC) database. Robotics Institute, p. 135 (2008)Google Scholar
  42. 42.
    Uijlings, J., Duta, I.C., Sangineto, E., Sebe, N.: Video classification with densely extracted HOG/HOF/MBH features: an evaluation of the accuracy/computational efficiency trade-off. Int. J. Multimed. Inf. Retr. 4(1), 33–44 (2015)CrossRefGoogle Scholar
  43. 43.
    Uijlings, J.R., Duta, I.C., Rostamzadeh, N., Sebe, N.: Realtime video classification using dense HOF/HOG. In: Proceedings of International Conference on Multimedia Retrieval, p. 145. ACM (2014)Google Scholar
  44. 44.
    Wang, H., Kläser, A., Schmid, C., Cheng-Lin, L.: Action recognition by dense trajectories. In: CVPR 2011-IEEE Conference on Computer Vision and Pattern Recognition, pp. 3169–3176. IEEE (2011)Google Scholar
  45. 45.
    Wang, X., Wu, Y., Zhu, L., Yang, Y.: Baidu-UTS submission to the EPIC-kitchens action recognition challenge 2019. arXiv preprint arXiv:1906.09383 (2019)
  46. 46.
    Wannenwetsch, A.S., Keuper, M., Roth, S.: ProbFlow: joint optical flow and uncertainty estimation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1182–1191. IEEE (2017)Google Scholar
  47. 47.
    Wrzalik, M., Krechel, D.: Human action recognition using optical flow and convolutional neural networks. In: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 801–805. IEEE (2017)Google Scholar
  48. 48.
    Xu, C., Chai, D., He, J., Zhang, X., Duan, S.: InnoHAR: a deep neural network for complex human activity recognition. IEEE Access 7, 9893–9902 (2019)CrossRefGoogle Scholar
  49. 49.
    Yordanova, K., Krüger, F.: Creating and exploring semantic annotation for behaviour analysis. Sensors 18(9), 2778 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Data and Web Science GroupUniversity of MannheimMannheimGermany
  2. 2.College of Science and EngineeringHamad Bin Khalifa UniversityDohaQatar

Personalised recommendations