A Powerful and Cost-Efficient Human Perception System for Camera Networks and Mobile Robotics

  • Marco CarraroEmail author
  • Matteo MunaroEmail author
  • Emanuele MenegattiEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 531)


In this work, we present a software library which enables the efficient use of the Kinect One, a time-of-flight RGB-D sensor, with the nVidia Jetson TK1, an ARM-based embedded system, for the purpose of people detection. Our software exploits nVidia CUDA to process all data necessary for robust people detection algorithm and other perception algorithms by parallelizing the generation of the 3D point cloud and many pixel-wise operations on both the raw depth and the infrared images coming from the Kinect One sensor. The library developed has been released as open-source and the whole system has been tested as a people detection node in an open source multi-node RGB-D tracking framework (OpenPTrack). The results gathered show that the proposed system can be effectively used as a people detection node, outperforming the state-of-the-art in terms of people detection frame rate not only with the nVidia Jetson, but also with non-embedded computers.


People detection and tracking Mobile robotics Kinect one nVidia Jetson OpenPTrack 



Portions of this work have been supported by NVidia, OpenPerception and the REMAP center at UCLA. The authors would like to thank Randy Illum and Jeff Burke at UCLA for the extensive testing of the developed software.


  1. 1.
    Zennaro, S., Munaro, M., Milani, S., Zanuttigh, P., Bernardi, A., Ghidoni, S., Menegatti, E.: Performance evaluation of the 1st and 2nd generation kinect for multimedia applications. In: 2015 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2015)Google Scholar
  2. 2.
    Gokturk, S.B., Yalcin, H., Bamji, C.: A time-of-flight depth sensor-system description, issues and solutions. In: Conference on Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04, pp. 35–35. IEEE (2004)Google Scholar
  3. 3.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5 (2009)Google Scholar
  4. 4.
    Munaro, M., Basso, F., Menegatti, E.: Openptrack: open source multi-camera calibration and people tracking for RGB-D camera networks. Robot. Auton. Syst. (2015)Google Scholar
  5. 5.
    Khoshelham, K., Elberink, S.O.: Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(2), 1437–1454 (2012)CrossRefGoogle Scholar
  6. 6.
    Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., Burgard, W.: An evaluation of the RGB-D slam system. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 1691–1696. IEEE (2012)Google Scholar
  7. 7.
    Munaro, M., Menegatti, E.: Fast RGB-D people tracking for service robots. Auton. Robots 37(3), 227–242 (2014)CrossRefGoogle Scholar
  8. 8.
    Munaro, M., Basso, F., Michieletto, S., Pagello, E., Menegatti, E.: A software architecture for RGB-D people tracking based on ROS framework for a mobile robot. In: Frontiers of Intelligent Autonomous Systems, pp. 53–68. Springer (2013)Google Scholar
  9. 9.
    Fleuret, F., Shitrit, H.B., Fua, P.: Re-identification for improved people tracking. In: Person Re-identification, pp. 309–330. Springer (2014)Google Scholar
  10. 10.
    Munaro, M., Ghidoni, S., Dizmen, D.T., Menegatti, E.: A feature-based approach to people re-identification using skeleton keypoints. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 5644–5651. IEEE (2014)Google Scholar
  11. 11.
    Nanni, L., Munaro, M., Ghidoni, S., Menegatti, E., Brahnam, S.: Ensemble of different approaches for a reliable person re-identification system. Appl. Comput. Inf. (2015)Google Scholar
  12. 12.
    Carraro, M., Antonello, M., Tonin, L., Menegatti, E.: An open source robotic platform for ambient assisted living. In: Artificial Intelligence and Robotics (AIRO) (2015)Google Scholar
  13. 13.
    Fischinger, D., Einramhof, P., Papoutsakis, K., Wohlkinger, W., Mayer, P., Panek, P., Hofmann, S., Koertner, T., Weiss, A., Argyros, A., et al.: Hobbit, a care robot supporting independent living at home: first prototype and lessons learned. Robot. Auton. Syst. (2014)Google Scholar
  14. 14.
    Ghidoni, S., Anzalone, S.M., Munaro, M., Michieletto, S., Menegatti, E.: A distributed perception infrastructure for robot assisted living. Robot. Auton. Syst. 62(9), 1316–1328 (2014)CrossRefGoogle Scholar
  15. 15.
    Basso, F., Pretto, A., Menegatti, E.: Unsupervised intrinsic and extrinsic calibration of a camera-depth sensor couple. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 6244–6249. IEEE (2014)Google Scholar
  16. 16.
    Nickolls, J., Buck, I., Garland, M., Skadron, K.: Scalable parallel programming with CUDA. Queue 6(2), 40–53 (2008)CrossRefGoogle Scholar
  17. 17.
    Xiang, L.: libfreenect2 CUDA library. (2015). [Online; accessed 2016-02-03]
  18. 18.
    Wang, X.: Intelligent multi-camera video surveillance: a review. Pattern Recognit. Lett. 34(1), 3–19 (2013)CrossRefGoogle Scholar
  19. 19.
    Vezzani, R., Baltieri, D., Cucchiara, R.: Pathnodes integration of standalone particle filters for people tracking on distributed surveillance systems. In: Image Analysis and Processing–ICIAP 2009, pp. 404–413. Springer (2009)Google Scholar
  20. 20.
    Kandhalu, A., Rowe, A., Rajkumar, R., Huang, C., Yeh, C.-C.: Real-time video surveillance over IEEE 802.11 mesh networks. In: Real-Time and Embedded Technology and Applications Symposium, 2009. RTAS 2009. 15th IEEE, pp. 205–214. IEEE (2009)Google Scholar
  21. 21.
    Jafari, O.H., Mitzel, D., Leibe, B.: Real-time RGB-D based people detection and tracking for mobile robots and head-worn cameras. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 5636–5643. IEEE (2014)Google Scholar
  22. 22.
    Munaro, M., Basso, F., Menegatti, E.: Tracking people within groups with RGB-D data. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2101–2107. IEEE (2012)Google Scholar
  23. 23.
    Munaro, M., Lewis, C., Chambers, D., Hvass, P., Menegatti, E.: RGB-D human detection and tracking for industrial environments. In: Intelligent Autonomous Systems, vol. 13, pp. 1655–1668. Springer (2016)Google Scholar
  24. 24.
    Sanders, J., Kandrot, E.: CUDA by example: an introduction to general-purpose GPU programming. Addison-Wesley Professional (2010)Google Scholar
  25. 25.
    Ryoo, S., Rodrigues, C.I., Baghsorkhi, S.S., Stone, S.S., Kirk, D.B., Hwu, W.-m.W.: Optimization principles and application performance evaluation of a multithreaded GPU using CUDA. In: Proceedings of the 13th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pp. 73–82. ACM (2008)Google Scholar
  26. 26.
    Aldoma, A., Tombari, F., Prankl, J., Richtsfeld, A., Di Stefano, L., Vincze, M.: Multimodal cue integration through hypotheses verification for RGB-D object recognition and 6DOF pose estimation. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 2104–2111. IEEE (2013)Google Scholar
  27. 27.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: using kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 31(5), 647–663 (2012)CrossRefGoogle Scholar
  28. 28.
    Carraro, M., Munaro, M., Menegatti, E.: Cost-efficient RGB-D smart camera for people detection and tracking. J. Electron. Imaging 25(4), 041007–041007 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PadovaPadovaItaly

Personalised recommendations