Skip to main content
SpringerLink
Log in

Embedded Deep Learning Solution for Person Identification and Following with a Robot

  • Conference paper
  • First Online:
Advances in Physical Agents II (WAF 2020)

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1285))

Included in the following conference series:

Abstract

This research presents a robust embedded system for following a person, making use of a pipeline of convolutional neural networks. Besides, it features an optical tracking system for supporting the inferences of the neural networks, allowing to determine the position of a person using an RGBD camera. The system is deployed using ROS, and runs in a NVIDIA Jetson TX2, an embedded SoM (System-on-Module), capable of performing computationally demanding tasks onboard, and coping with the complexity required to run a robust tracking and following algorithm. The board is attached to a robotic mobile base, which receives velocity commands to move the system towards the target person.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The tiny version of YOLOv3 is used to the limited memory on the Jetson TX2 board. The full YOLOv3 model requires more memory than the available size.

  2. 2.

    https://github.com/mystic123/tensorflow-yolo-v3.

  3. 3.

    https://github.com/davidsandberg/facenet.

  4. 4.

    https://developer.nvidia.com/tensorrt.

  5. 5.

    https://github.com/wkentaro/labelme.

  6. 6.

    https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md.

  7. 7.

    https://www.youtube.com/watch?v=WZ0riKMwJWA.

References

  1. Pennisi, P., Tonacci, A., Tartarisco, G., Billeci, L., Ruta, L., Gangemi, S., Pioggia, G.: Autism and social robotics: a systematic review, Autism Research (2016)

    Google Scholar 

  2. Cheong, A., et al.: Development of a robotic waiter system,ll IFAC-PapersOnLine, In: 7th IFAC Symposium on Mechatronic Systems MECHATRONICS 2016, vol. 49, no. 21, pp. 681–686 (2016). https://doi.org/10.1016/j.ifacol.2016.10.679, http://www.sciencedirect.com/science/article/pii/S2405896316322911

  3. Gockley, R., Forlizzi, J., Simmons, R.: Natural person-following behavior for social robots. Association for Computing Machinery, New York (2007)

    Google Scholar 

  4. Condés, I., Cañas, J.-M.: Person Following Robot Behavior Using Deep Learning: Proceedings of the 19th International Workshop of Physical Agents (WAF 2018), November 22-23, 2018, Madrid, Spain, pp. 147–161 (2019)

    Google Scholar 

  5. Sidenbladh, H., Kragic, D., Christensen, H.I.: A person following behaviour for a mobile robot. In: 1999 IEEE International Conference on Robotics and Automation, 1999. Proceedings. vol. 1, pp. 670–675. IEEE (1999)

    Google Scholar 

  6. Yoshimi, T., Nishiyama, M., Sonoura, T., Nakamoto, H., Tokura, S., Sato, H., Ozaki, F., Matsuhira, N., Mizoguchi, H.: Development of a person following robot with vision based target detection, In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5286–5291. IEEE (2006)

    Google Scholar 

  7. Aguirre, E., García-Silvente, M., Pascual, D.: A multisensor based approach using supervised learning and particle filtering for people detection and tracking. In: Robot : Second Iberian Robotics Conference, vol. 2016, pp. 645–657 Springer, Cham (2015)

    Google Scholar 

  8. Muñoz-Salinas, R., Aguirre, E., García-Silvente, M.: People detection and tracking using stereo vision and color. Image Vision Comput. 25(6), 995–1007 (2007)

    Article  Google Scholar 

  9. Satake, J., Miura, J.: Robust stereo-based person detection and tracking for a person following robot. In: ICRA Workshop on People Detection and Tracking, pp. 1–10 (2009)

    Google Scholar 

  10. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005) (2005)

    Google Scholar 

  11. Ilias, B., Shukor, S.A., Yaacob, S., Adom, A., Razali, M.M.: A nurse following robot with high speed Kinect sensor. ARPN J. Eng. Appl. Sci. 9(12), 2454–2459 (2014)

    Google Scholar 

  12. Shimura, K., Ando, Y., Yoshimi, T., Mizukawa, M.: Research on person following system based on RGB-D features by autonomous robot with multi-kinect sensor. In: 2014 IEEE/SICE International Symposium on System Integration (SII) (2014)

    Google Scholar 

  13. Xue, H., Liu, Y., Cai, D., He, X.: Tracking people in RGBD videos using deep learning and motion clues. Neurocomputing 204, 70–76 (2016)

    Article  Google Scholar 

  14. Kouno, D., Shimada, K., Endo, T.: Person identification using top-view image with depth information. In: 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (2012)

    Google Scholar 

  15. You, J., Wu, A., Li, X., Zheng, W.-S.: Top-push video-based person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  16. Yoon, Y., Yoon, H., Kim, J.: Person reidentification in a person-following robot. In: 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN) (2016)

    Google Scholar 

  17. Munaro, M., Ghidoni, S., Dizmen, D.T., Menegatti, E.: A feature-based approach to people re-identification using skeleton keypoints. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 5644–5651. IEEE (2014)

    Google Scholar 

  18. Koide, K., Miura, J.: Identification of a specific person using color, height, and gait features for a person following robot. Robot. Autonomous Syst. 84, 76–87 (2016)

    Article  Google Scholar 

  19. Welsh, J.B.: Real-time pose based human detection and re-identification with a single camera for robot person following, Ph.D. dissertation, University of Maryland, College Park (2017)

    Google Scholar 

  20. Ahmed, E., Jones, M., Marks, T.K.: An improved deep learning architecture for person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  21. McLaughlin, N., Martinez del Rincon, J., Miller, P.: Recurrent convolutional network for video-based person re-identification, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  22. “SSD: Single Shot MultiBox Detector.”

    Google Scholar 

  23. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications (2017)

    Google Scholar 

  24. Redmon, J., Farhadi, A.: YOLOv3: An Incremental Improvement (2018)

    Google Scholar 

  25. ModelZoo: TensorFlow Object Detection: Model Zoo. https://modelzoo.co/model/objectdetection

  26. Itzcovich, I.: Faced: CPU Real Time face detection using Deep Learning (2005)

    Google Scholar 

  27. Redmon, J., Farhadi, A.: Yolo9000: Better, faster, stronger (2016)

    Google Scholar 

  28. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features, vol. 1, pp. I–511 (2001)

    Google Scholar 

  29. “FaceNet: A unified embedding for face recognition and clustering.”

    Google Scholar 

  30. González-Díaz, I.: Computer Vision: Dense Motion Estimation (2020)

    Google Scholar 

  31. Lucas, B., Kanade, T.: An Iterative Image Registration Technique with an Application to Stereo Vision (IJCAI), vol. 81, April 1981

    Google Scholar 

  32. Shi, J., Tomasi, U.: Good features to track. In: 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (1994)

    Google Scholar 

  33. GoogleAIBlog, “Accelerating Training and Inference with the Tensorflow Object Detection API.” https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html

Download references

Author information

Authors and Affiliations

  1. Universidad Carlos III de Madrid, Leganés, Madrid, Spain

    Ignacio Condés

  2. Universidad Rey Juan Carlos, Fuenlabrada, Madrid, Spain

    José-María Cañas & Eduardo Perdices

Authors
  1. Ignacio Condés
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. José-María Cañas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eduardo Perdices
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ignacio Condés .

Editor information

Editors and Affiliations

  1. Electronics Department, University of Alcalá, Madrid, Spain

    Luis M. Bergasa

  2. Electronics Department, University of Alcalá, Madrid, Spain

    Manuel Ocaña

  3. Electronics Department, University of Alcalá, Madrid, Spain

    Rafael Barea

  4. Electronics Department, University of Alcalá, Madrid, Spain

    Elena López-Guillén

  5. Electronics Department, University of Alcalá, Madrid, Spain

    Pedro Revenga

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Condés, I., Cañas, JM., Perdices, E. (2021). Embedded Deep Learning Solution for Person Identification and Following with a Robot. In: Bergasa, L.M., Ocaña, M., Barea, R., López-Guillén, E., Revenga, P. (eds) Advances in Physical Agents II. WAF 2020. Advances in Intelligent Systems and Computing, vol 1285. Springer, Cham. https://doi.org/10.1007/978-3-030-62579-5_20

Download citation

Publish with us

Policies and ethics

Search

Navigation