Robot Vision System for Real-Time Human Detection and Action Recognition

  • Satoshi HoshinoEmail author
  • Kyohei Niimura
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 867)


Mobile robots equipped with camera sensors are required to perceive surrounding humans and their actions for safe autonomous navigation. These are so-called human detection and action recognition. In this paper, moving humans are target objects. Compared to computer vision, the real-time performance of robot vision is more important. For this challenge, we propose a robot vision system. In this system, images described by the optical flow are used as an input. For the classification of humans and actions in the input images, we use Convolutional Neural Network, CNN, rather than coding invariant features. Moreover, we present a novel detector, local search window, for clipping partial images around target objects. Through the experiment, finally, we show that the robot vision system is able to detect the moving human and recognize the action in real time.


Robot vision Real-time image processing CNN Optical flow 


  1. 1.
    Ojala, T., et al.: Performance evaluation of texture measures with classification based on Kullback discrimination of distributions. In: International Conference on Pattern Recognition, vol. 1, pp. 582–585 (1994)Google Scholar
  2. 2.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: International Conference on Computer Vision, pp. 1150–1157 (1999)Google Scholar
  3. 3.
    Csurka, G., et al.: Visual categorization with bags of keypoints. In: International Workshop on Statistical Learning in Computer Vision, pp. 59–74 (2004)Google Scholar
  4. 4.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)Google Scholar
  5. 5.
    Dollar, P., et al.: Behavior recognition via sparse spatio-temporal features. In: International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pp. 65–72 (2005)Google Scholar
  6. 6.
    van de Sande, K.E.A., et al.: Segmentation as selective search for object recognition. In: IEEE International Conference on Computer Vision, pp. 1879–1886 (2011)Google Scholar
  7. 7.
    Uijlings, J.R.R., et al.: Selective search for object recognition. In: International Journal of Computer Vision, vol. 104, pp. 154–171 (2013)CrossRefGoogle Scholar
  8. 8.
    LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  9. 9.
    Ren, S., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 36(6), 1137–1149 (2016)CrossRefGoogle Scholar
  10. 10.
    Farneb\({\rm \ddot{a}}\)ck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian Conference on Image Analysis, vol. 2749, pp. 363–370 (2003)Google Scholar
  11. 11.
    Fathi, A., Mori, G.: Action recognition by learning mid-level motion features. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2749, pp. 1–8 (2008)Google Scholar
  12. 12.
    Jain, M., et al.: Better exploiting motion for better action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2555–2562 (2013)Google Scholar
  13. 13.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: International Conference for Learning Representations (2015)Google Scholar
  14. 14.
    Srivastava, N., et al.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  15. 15.
    LeCun, Y., et al.: Deep learning. Nature 521(7553), 436–444 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)CrossRefGoogle Scholar
  17. 17.
    Goudail, F., et al.: Bhattacharyya distance as a contrast parameter for statistical processing of noisy optical images. J. Opt. Soc. Am. A 21(7), 1231–1240 (2004)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Comaniciu, D., et al.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)CrossRefGoogle Scholar
  19. 19.
    Oliveira, L., et al.: On exploration of classifier ensemble synergism in pedestrian detection. IEEE Trans. Intell. Transp. Syst. 11(1), 16–27 (2010)CrossRefGoogle Scholar
  20. 20.
    Wang, H., Schmid, C.: LEAR-INRIA submission for the THUMOS workshop. In: ICCV Workshop on Action Recognition with a Large Number of Classes, vol. 2, no. 7 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Mechanical and Intelligent EngineeringUtsunomiya UniversityUtsunomiyaJapan

Personalised recommendations