Abstract
In this paper, we introduce a stereo vision based CNN tracker for a person following robot. The tracker is able to track a person in real-time using an online convolutional neural network. Our approach enables the robot to follow a target under challenging situations such as occlusions, appearance changes, pose changes, crouching, illumination changes or people wearing the same clothes in different environments. The robot follows the target around corners even when it is momentarily unseen by estimating and replicating the local path of the target. We build an extensive dataset for person following robots under challenging situations. We evaluate the proposed system quantitatively by comparing our tracking approach with existing real-time tracking algorithms.
B.X. Chen and R. Sahdev—Denotes equal contribution.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Ferreira, B.Q., Karipidou, K., Rosa, F., Petisca, S., Alves-Oliveira, P., Paiva, A.: A study on trust in a robotic suitcase. In: Agah, A., Cabibihan, J.-J., Howard, A.M., Salichs, M.A., He, H. (eds.) ICSR 2016. LNCS, vol. 9979, pp. 179–189. Springer, Cham (2016). doi:10.1007/978-3-319-47437-3_18
Awai, M., Shimizu, T., Kaneko, T., Yamashita, A., Asama, H.: Hog-based person following and autonomous returning using generated map by mobile robot equipped with camera and laser range finder. In: Lee, S., Cho, H., Yoon, KJ., Lee J. (eds.) Intelligent Autonomous Systems 12, Advances in Intelligent Systems and Computing, vol. 194, pp. 51–60. Springer, Heidelberg (2013). doi:10.1007/978-3-642-33932-5_6
Borenstein, J., Feng, L.: Umbmark: a benchmark test for measuring odometry errors in mobile robots. In: Photonics East 1995, pp. 113–124. International Society for Optics and Photonics (1995)
Calisi, D., Iocchi, L., Leone, R.: Person following through appearance models and stereo vision using a mobile robot. In: VISApp Workshop on Robot Vision, pp. 46–56 (2007)
Camplani, M., Hannuna, S.L., Mirmehdi, M., Damen, D., Paiement, A., Tao, L., Burghardt, T.: Real-time RGB-D tracking with depth scaling kernelised correlation filters and occlusion handling. In: British Machine Vision Conference, Swansea, UK, 7–10 September 2015. BMVA Press (2015)
Chen, B.X., Sahdev, R., Tsotsos, J.K.: Person following robot using selected online Ada-boosting with stereo camera. In: 2017 14th Conference on Computer and Robot Vision (CRV), pp. 48–55. IEEE (2017)
Chen, Z., Birchfield, S.T.: Person following with a mobile robot using binocular feature-based tracking. In: IROS 2007. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 815–820. IEEE (2007)
Chivilò, G., Mezzaro, F., Sgorbissa, A., Zaccaria, R.: Follow-the-leader behaviour through optical flow minimization. In: Proceedings of 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2004, vol. 4, pp. 3182–3187. IEEE (2004)
Cosgun, A., Florencio, D.A., Christensen, H.I.: Autonomous person following for telepresence robots. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 4335–4342. IEEE (2013)
Couprie, C., Farabet, C., Najman, L., Lecun, Y.: Indoor semantic segmentation using depth information. In: International Conference on Learning Representations (ICLR 2013), April 2013 (2013)
Danelljan, M., Häger, G., Khan, F., Felsberg, M.: Accurate scale estimation for robust visual tracking. In: British Machine Vision Conference, Nottingham, 1–5 September 2014. BMVA Press (2014)
Doisy, G., Jevtic, A., Lucet, E., Edan, Y.: Adaptive person-following algorithm based on depth images and mapping. In: Proceedings of the IROS Workshop on Robot Motion Planning (2012)
Eitel, A., Springenberg, J.T., Spinello, L., Riedmiller, M., Burgard, W.: Multimodal deep learning for robust RGB-D object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 681–687. IEEE (2015)
Fan, J., Xu, W., Wu, Y., Gong, Y.: Human tracking using convolutional neural networks. IEEE Trans. Neural Netw. 21(10), 1610–1623 (2010)
Fuentes-Pacheco, J., Ruiz-Ascencio, J., Rendón-Mancha, J.M.: Visual simultaneous localization and mapping: a survey. Artif. Intell. Rev. 43(1), 55–81 (2015)
Gao, C., Chen, F., Yu, J.G., Huang, R., Sang, N.: Robust visual tracking using exemplar-based detectors. IEEE Trans. Circuits Syst. Video Technol. 27(2), 300–312 (2015)
Gao, C., Shi, H., Yu, J.G., Sang, N.: Enhancement of elda tracker based on cnn features and adaptive model update. Sensors 16(4), 545 (2016)
Grabner, H., Grabner, M., Bischof, H.: Real-time tracking via on-line boosting. In: Proceedings of the British Machine Vision Conference 2006, Edinburgh, pp. 47–56 (2006)
Gupta, S., Girshick, R., Arbeláez, P., Malik, J.: Learning rich features from RGB-D images for object detection and segmentation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 345–360. Springer, Cham (2014). doi:10.1007/978-3-319-10584-0_23
Hare, S., Golodetz, S., Saffari, A., Vineet, V., Cheng, M.M., Hicks, S.L., Torr, P.H.: Struck: structured output tracking with kernels. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 2096–2109 (2016)
Hong, S., You, T., Kwak, S., Han, B.: Online tracking by learning discriminative saliency map with convolutional neural network. In: ICML, pp. 597–606 (2015)
Hua, Y., Alahari, K., Schmid, C.: Online object tracking with proposal selection. In: The IEEE International Conference on Computer Vision (ICCV), December 2015
Kanbara, M., Okuma, T., Takemura, H., Yokoya, N.: A stereoscopic video see-through augmented reality system based on real-time vision-based registration. In: Proceedings of IEEE Virtual Reality, pp. 255–262. IEEE (2000)
Kobilarov, M., Sukhatme, G., Hyams, J., Batavia, P.: People tracking and following with mobile robot using an omnidirectional camera and a laser. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, ICRA 2006, pp. 557–562. IEEE (2006)
Koide, K., Miura, J.: Identification of a specific person using color, height, and gait features for a person following robot. Robot. Auton. Syst. 84, 76–87 (2016)
Nishimura, S., Itou, K., Kikuchi, T., Takemura, H., Mizoguchi, H.: A study of robotizing daily items for an autonomous carrying system-development of person following shopping cart robot. In: 9th International Conference on Control, Automation, Robotics and Vision, ICARCV 2006, pp. 1–6. IEEE (2006)
O’Dwyer, A.: Handbook of PI and PID Controller Tuning Rules. World Scientific, Singapore (2009)
Oron, S., Bar-Hillel, A., Levi, D., Avidan, S.: Locally orderless tracking. Int. J. Comput. Vis. 111(2), 213–228 (2015)
Sardari, F., Moghaddam, M.E.: A hybrid occlusion free object tracking method using particle filter and modified galaxy based search meta-heuristic algorithm. Appl. Soft Comput. 50, 280–299 (2017)
Satake, J., Chiba, M., Miura, J.: A sift-based person identification using a distance-dependent appearance model for a person following robot. In: 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 962–967. IEEE (2012)
Schlegel, C., Jaberg, H., Schuster, M.: Vision based person tracking with a mobile robot. In: Proceedings of British Machine Vision Conference. Citeseer (1998)
Song, S., Xiao, J.: Tracking revisited using RGBD camera: unified benchmark and baselines. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 233–240 (2013)
Takemura, H., Ito, K., Mizoguchi, H.: Person following mobile robot under varying illumination based on distance and color information. In: IEEE International Conference on Robotics and Biomimetics, ROBIO 2007, pp. 1500–1505. IEEE (2007)
Tarokh, M., Ferrari, P.: Case study: robotic person following using fuzzy control and image segmentation. J. Field Robot. 20(9), 557–568 (2003)
Wu, Y., Lim, J., Yang, M.H.: Object tracking benchmark. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1834–1848 (2015)
Yamane, T., Shirai, Y., Miura, J.: Person tracking by integrating optical flow and uniform brightness regions. In: Proceedings of 1998 IEEE International Conference on Robotics and Automation, vol. 4, pp. 3267–3272. IEEE (1998)
Yoshimi, T., Nishiyama, M., Sonoura, T., Nakamoto, H., Tokura, S., Sato, H., Ozaki, F., Matsuhira, N., Mizoguchi, H.: Development of a person following robot with vision based target detection. In: 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5286–5291. IEEE (2006)
Zhai, M., Roshtkhari, M.J., Mori, G.: Deep learning of appearance models for online object tracking. arXiv preprint arXiv:1607.02568 (2016)
Zhang, K., Zhang, L., Yang, M.H.: Real-time object tracking via online discriminative feature selection. IEEE Trans. Image Process. 22(12), 4664–4677 (2013)
Zhang, L., Suganthan, P.N.: Visual tracking with convolutional neural network. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2072–2077. IEEE (2015)
Zhang, L., van der Maaten, L.: Structure preserving object tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1838–1845 (2013)
Acknowledgement
We acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the NSERC Strategic Network for Field Robotics (NCFRN), and the Canada Research Chairs Program through grants to John K. Tsotsos. The authors would like to thank Sidharth Sahdev for helping in the process of dataset generation and making the video for this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Chen, B.X., Sahdev, R., Tsotsos, J.K. (2017). Integrating Stereo Vision with a CNN Tracker for a Person-Following Robot. In: Liu, M., Chen, H., Vincze, M. (eds) Computer Vision Systems. ICVS 2017. Lecture Notes in Computer Science(), vol 10528. Springer, Cham. https://doi.org/10.1007/978-3-319-68345-4_27
Download citation
DOI: https://doi.org/10.1007/978-3-319-68345-4_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-68344-7
Online ISBN: 978-3-319-68345-4
eBook Packages: Computer ScienceComputer Science (R0)