Skip to main content
SpringerLink
Log in

Simultaneous people tracking and robot localization in dynamic social spaces

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Accurate robot localization and people tracking are necessary for deploying service robots in crowded everyday environments such as shopping malls, and features like product displays change over time, making map-based localization using on-board sensors difficult. We propose the use of an external sensor system to track people together with one or more robots. This approach is more robust to occlusions than on-board sensing and is unaffected by changing map features. In our system, laser range finders track people and robots in the environment, and odometry data is used to associate each robot with a tracked entity and correct the robot’s pose. Techniques are also presented for identifying and recovering from tracking errors. Simulation results show that our system can outperform localization using on-board sensors, both in tracking accuracy and in automatic recovery from errors. We demonstrate our system’s effectiveness in simulation, in a controlled experiment in a real shopping mall environment, and in real human–robot interactions with customers in a busy shopping arcade.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://software-and-algorithms.blogspot.jp/2012/09/the-hungarian-algorithm-for-assignment.html

  2. 3DTK — The 3D Toolkit. http://slam6d.sourceforge.net/. Accessed July 22, 2013

References

  • Amarasinghe, D., Mann, G. K. I., & Gosine, R.G. (2008). Integrated laser-camera sensor for the detection and localization of landmarks for robotic applications. In IEEE International Conference on Robotics and Automation, 2008. ICRA 2008, 19–23 May 2008 (pp. 4012–4017). doi:10.1109/robot.2008.4543827.

  • Bennewitz, M., Burgard, W., Cielniak, G., & Thrun, S. (2005). Learning motion patterns of people for compliant robot motion. The International Journal of Robotics Research, 24(1), 31–48.

    Article  Google Scholar 

  • Bevilacqua, A., Di Stefano, L., & Azzari, P. (2006). People tracking using a time-of-flight depth sensor. In IEEE International Conference on Video and Signal Based Surveillance, 2006. AVSS ’06, Nov. 2006 (pp. 89–89). doi:10.1109/avss.2006.92.

  • Billard, A., Ijspeert, A. J., & Martinoli, A. (1999). A multi-robot system for adaptive exploration of a fast-changing environment: Probabilistic modeling and experimental study. Connection Science, 11(3–4), 359–379. doi:10.1080/095400999116304.

    Article  Google Scholar 

  • Blackman, S. S. (2004). Multiple hypothesis tracking for multiple target tracking. Aerospace and Electronic Systems Magazine, IEEE, (Vol. 19, pp. 5–18). doi:10.1109/MAES.2004.1263228

  • Borrmann, D., Elseberg, J., Lingemann, K., Nüchter, A., & Hertzberg, J. (2008). Globally consistent 3D mapping with scan matching. Robotics and Autonomous Systems, 56(2), 130–142. doi:10.1016/j.robot.2007.07.002.

    Article  Google Scholar 

  • Brščić, D., Kanda, T., Ikeda, T., & Miyashita, T. (2013). Person tracking in large public spaces using 3-D range sensors. IEEE Transactions on Human-Machine Systems, 43(6), 522–534. doi:10.1109/thms.2013.2283945.

    Article  Google Scholar 

  • Burgard, W., Cremers, A. B., Fox, D., Hahnel, D., Lakemeyer, G., & Schulz, D., et al. (1998). The interactive museum tour-guide robot. In Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, Madison, Wisconsin, United States, (pp. 11–18).

  • Burgard, W., Moors, M., Stachniss, C., & Schneider, F. E. (2005). Coordinated multi-robot exploration. IEEE Transactions on Robotics, 21(3), 376–386. doi:10.1109/tro.2004.839232.

    Article  Google Scholar 

  • Cui, J., Zha, H., Zhao, H., & Shibasaki, R. (2007). Laser-based detection and tracking of multiple people in crowds. Computer Vision and Image Understanding, 106(2–3), 300–312. doi:10.1016/j.cviu.2006.07.015.

    Article  Google Scholar 

  • Dissanayake, M. W. M. G., Newman, P., Clark, S., Durrant-Whyte, H. F., & Csorba, M. (2001). A solution to the simultaneous localization and map building (SLAM) problem. IEEE Transactions on Robotics and Automation, 17(3), 229–241. doi:10.1109/70.938381.

    Article  Google Scholar 

  • Elfes, A. (1989). Using occupancy grids for mobile robot perception and navigation. Computer, 22(6), 46–57. doi:10.1109/2.30720.

    Article  Google Scholar 

  • Fod, A., Howard, A., & Mataric, M.A.J. (2002). A laser-based people tracker. In Proceedings ICRA ’02. IEEE International Conference on Robotics and Automation, 2002, (Vol. 3, pp. 3024–3029). doi:10.1109/robot.2002.1013691.

  • Fox, D. (2003). Adapting the sample size in particle filters through KLD-sampling. The International Journal of Robotics Research, 22, 985–1004. doi:10.1177/0278364903022012001.

    Article  Google Scholar 

  • Fox, D., Burgard, W., & Thrun, S. (1999). Markov localization for mobile robots in dynamic environments. Journal of Artificial Intelligence Research, 2, 391–327.

    Google Scholar 

  • Gerkey, B., Vaughan, R. T., & Howard, A. (2003). The player/stage project: Tools for multi-robot and distributed sensor systems. In 11th International Conference on Advanced Robotics (ICAR 2003), Coimbra, Portugal, June 2003 (pp. 317–323).

  • Glas, D. F., Ferreri, F., Miyashita, T., Ishiguro, H., & Hagita, N. (2012a). Automatic calibration of laser range finder positions for pedestrian tracking based on social group detections. Advanced Robotics, 28(9), 573–588. doi:10.1080/01691864.2013.879272.

    Google Scholar 

  • Glas, D. F., Kanda, T., Ishiguro, H., & Hagita, N. (2009a). Simultaneous people tracking and localization for social robots using external laser range finders. In Intelligent Robots and Systems (IROS), IEEE/RSJ International Conference on, St. Louis, MO, USA, 10–15 Oct. 2009 (pp. 846–853). doi:10.1109/iros.2009.5354198.

  • Glas, D. F., Kanda, T., Ishiguro, H., & Hagita, N. (2012b). Teleoperation of multiple social robots. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 42(3), 530–544. doi:10.1109/tsmca.2011.2164243.

    Article  Google Scholar 

  • Glas, D. F., Miyashita, T., Ishiguro, H., & Hagita, N. (2009b). Laser-based tracking of human position and orientation using parametric shape modeling. Advanced Robotics, 23(4), 405–428. doi:10.1163/156855309x408754.

    Article  Google Scholar 

  • Glas, D. F., Miyashita, T., Ishiguro, H., & Hagita, N. (2010). Automatic position calibration and sensor displacement detection for networks of laser range finders for human tracking. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 18–22 Oct. 2010 (pp. 2938–2945). doi:10.1109/iros.2010.5652272.

  • Gross, H.-M., Boehme, H., Schroeter, C., Müller, S., Koenig, A., & Einhorn, E., et al. (2009). TOOMAS: interactive shopping guide robots in everyday use-final implementation and experiences from long-term field trials. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009. IROS 2009, (pp. 2005–2012): IEEE.

  • Iwamura, Y., Shiomi, M., Kanda, T., Ishiguro, H., & Hagita, N. (2011). Do elderly people prefer a conversational humanoid as a shopping assistant partner in supermarkets? In Proceedings of the 6th International Conference on Human-Robot Interaction, Lausanne, Switzerland (pp. 449–456). doi:10.1145/1957656.1957816.

  • Kanda, T., Glas, D. F., Shiomi, M., & Hagita, N. (2009). Abstracting people’s trajectories for social robots to proactively approach customers. IEEE Transactions on Robotics, 25(6), 1382–1396. doi:10.1109/tro.2009.2032969.

    Article  Google Scholar 

  • Kanda, T., Hirano, T., Eaton, D., & Ishiguro, H. (2004a). Interactive robots as social partners and peer tutors for children: A field trial. Human-Computer Interaction, 19(1), 61–84. doi:10.1207/s15327051hci1901&2_4.

  • Kanda, T., Ishiguro, H., Imai, M., & Ono, T. (2004). Development and evaluation of interactive humanoid robots. Proceedings of the IEEE, 92(11), 1839–1850. doi:10.1109/jproc.2004.835359.

    Article  Google Scholar 

  • Kanda, T., Shiomi, M., Miyashita, Z., Ishiguro, H., & Hagita, N. (2010). A communication robot in a shopping mall. IEEE Transactions on Robotics, 26(5), 897–913. doi:10.1109/tro.2010.2062550.

    Article  Google Scholar 

  • Köse, H., & Akın, H. L. (2007). The reverse monte carlo localization algorithm. Robotics and Autonomous Systems, 55(6), 480–489. doi:10.1016/j.robot.2006.12.007.

    Article  Google Scholar 

  • Lee, J.-S., & Chung, W. K. (2010). Robust mobile robot localization in highly non-static environments. Autonomous Robots, 29(1), 1–16. doi:10.1007/s10514-010-9184-1.

    Article  Google Scholar 

  • Montemerlo, M., Thrun, S., & Whittaker, W. (2002). Conditional particle filters for simultaneous mobile robot localization and people-tracking. In Proceedings ICRA ’02. IEEE International Conference on Robotics and Automation, 2002 (Vol. 1, pp. 695–701 vol. 691). doi:10.1109/robot.2002.1013439.

  • Moravec, H., & Elfes, A. (1985). High resolution maps from wide angle sonar. In Proceedings 1985 IEEE International Conference on Robotics and Automation, Mar 1985 (Vol. 2, pp. 116–121). doi:10.1109/robot.1985.1087316.

  • Mutlu, B., & Forlizzi, J. (2008). Robots in organizations: the role of workflow, social, and environmental factors in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human robot interaction, Amsterdam, The Netherlands (pp. 287–294). doi:10.1145/1349822.1349860.

  • Ouellette, R., & Hirasawa, K. (2007). A comparison of SLAM implementations for indoor mobile robots. In Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on, Oct. 29 2007-Nov. 2 2007 (pp. 1479–1484). doi:10.1109/iros.2007.4399575.

  • Pacchierotti, E., Christensen, H. I., & Jensfelt, P. (2006). Design of an office-guide robot for social interaction studies. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006, 9–15 Oct. 2006 (pp. 4965–4970). doi:10.1109/iros.2006.282519.

  • Park, S., Saegusa, R., & Hashimoto, S. (2007). Autonomous navigation of a mobile robot based on passive RFID. In Robot and Human interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on, 26–29 Aug. 2007 (pp. 218–223). doi:10.1109/roman.2007.4415083.

  • Pizarro, D., Marron, M., Peon, D., Mazo, M., Garcia, J. C., & Sotelo, M. A., et al. (2008). Robot and obstacles localization and tracking with an external camera ring. In IEEE International Conference on Robotics and Automation, 2008. ICRA 2008, 19–23 May 2008 (pp. 516–521). doi:10.1109/robot.2008.4543259.

  • Reid, D. B. (1979). An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control (Vol. 24, pp. 843–854). doi:10.1109/TAC.1979.1102177

  • Saffiotti, A., Broxvall, M., Gritti, M., LeBlanc, K., Lundh, R., & Rashid, J., et al. (2008). The PEIS-Ecology project: Vision and results. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2008. IROS 2008, 22–26 Sept. 2008 (pp. 2329–2335). doi:10.1109/iros.2008.4650962.

  • Saiki, L. Y. M., Satake, S., Kanda, T., & Hagita, N. (2011). Modeling environments from a route perspective. In Proceedings of the 6th international conference on Human-robot interaction, Lausanne, Switzerland (pp. 441–448). doi:10.1145/1957656.1957815.

  • Satake, S., Kanda, T., Glas, D. F., Imai, M., Ishiguro, H., & Hagita, N. (2010). How to approach humans?: Strategies for social robots to initiate interaction. Journal of the Robotics Society of Japan, 28(3), 327–337.

    Article  Google Scholar 

  • Schulz, D., Burgard, W., Fox, D., & Cremers, A. B. (2003). People tracking with mobile robots using sample-based joint probabilistic data association filters. The International Journal of Robotics Research, 22(2), 99–116. doi:10.1177/0278364903022002002.

    Article  Google Scholar 

  • Schulz, D., Fox, D., & Hightower, J. (2003). People tracking with anonymous and ID-sensors using Rao-Blackwellised particle filters. In International Joint Conference on Artificial Intelligence (Vol. 18, pp. 921–928).

  • Siegwart, R., Arras, K. O., Bouabdallah, S., Burnier, D., Froidevaux, G., Greppin, X., et al. (2003). Robox at Expo. 02: A large-scale installation of personal robots. Robotics and Autonomous Systems, 42(3), 203–222. doi:10.1016/s0921-8890(02)00376-7.

    Article  MATH  Google Scholar 

  • Sisbot, E. A., Alami, R., Simeon, T., Dautenhahn, K., Walters, M., & Woods, S. (2005). Navigation in the presence of humans. In 5th IEEE-RAS International Conference on Humanoid Robots, 2005, 5–5 Dec. 2005 (pp. 181–188). doi:10.1109/ichr.2005.1573565.

  • Takahashi, T. (2007). 2D localization of outdoor mobile robots using 3D laser range data. Master’s Thesis, Carnegie Mellon University.

  • Thrun, S., Bücken, A., Burgard, W., Fox, D., Fröhlinghaus, T., Hennig, D., et al. (1998). Map learning and high-speed navigation in RHINO. In Artificial intelligence and mobile robots (pp. 21–52): MIT Press.

  • Thrun, S., Fox, D., Burgard, W., & Dallaert, F. (2001). Robust Monte Carlo localization for mobile robots. Artificial Intelligence, 128(1–2), 99–141. doi:10.1016/s0004-3702(01)00069-8.

    Article  MATH  Google Scholar 

  • Urmson, C., Anhalt, J., Bagnell, D., Baker, C., Bittner, R., Clark, M., et al. (2008). Autonomous driving in urban environments: Boss and the urban challenge. Journal of Field Robotics, 25(8), 425–466.

    Article  Google Scholar 

  • Vu, T.-D., Aycard, O., & Appenrodt, N. (2007). Online localization and mapping with moving object tracking in dynamic outdoor environments. In IEEE Intelligent Vehicles Symposium, 2007, 13–15 June 2007 (pp. 190–195). doi:10.1109/ivs.2007.4290113.

  • Wang, C.-C., Thorpe, C., & Thrun, S. (2003). Online simultaneous localization and mapping with detection and tracking of moving objects: theory and results from a ground vehicle in crowded urban areas. In Proceedings ICRA ’03. IEEE International Conference on Robotics and Automation, 2003, 14–19 Sept. 2003 (Vol. 1, pp. 842–849 vol. 841). doi:10.1109/robot.2003.1241698.

  • Wolf, D. F., & Sukhatme, G. S. (2005). Mobile robot simultaneous localization and mapping in dynamic environments. Autonomous Robots, 19(1), 53–65. doi:10.1007/s10514-005-0606-4.

    Article  Google Scholar 

  • Zhao, H., Chiba, M., Shibasaki, R., Shao, X., Cui, J., & Zha, H. (2008). SLAM in a dynamic large outdoor environment using a laser scanner. In IEEE International Conference on Robotics and Automation, 2008. ICRA 2008, 19–23 May 2008 (pp. 1455–1462). doi:10.1109/robot.2008.4543407.

Download references

Acknowledgments

Thanks to the staff of Universal CityWalk Osaka and Apita Town Keihanna for their cooperation in this research, and thanks to Dr. Satoshi Koizumi for his assistance in organizing the field trials. This research was supported in part by the Ministry of Internal Affairs and Communications of Japan, and in part by JST, CREST.

Author information

Authors and Affiliations

  1. ATR-IRC, 2-2-2 Hikaridai, Keihanna Science City, Kyoto, Japan

    Dylan F. Glas, Yoichi Morales, Takayuki Kanda & Norihiro Hagita

  2. IRL, Osaka University, 1-3 Machikaneyama, Toyonaka, Osaka, Japan

    Hiroshi Ishiguro

Authors
  1. Dylan F. Glas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yoichi Morales
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Takayuki Kanda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hiroshi Ishiguro
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Norihiro Hagita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dylan F. Glas.

Appendix

Appendix

Most mapping and localization approaches for mobile robots rely on the assumption that environments are static and do not change over time. Many dynamic obstacles such as pedestrians were present in our shopping mall environment, necessitating a method for removing moving objects from the robot’s scan data. The next subsection explains this approach.

1.1 Obstacle removal

First we process each scan of laser data, grouping points into clusters. When two consecutive points are within 0.1 m of distance they are grouped as a cluster. A covariance matrix is built from the point distribution of each cluster and the eigenvalues of the matrix are computed to analyze the shape of the cluster. If the length of a cluster is smaller than 0.5 m and the cluster distribution does not represent a straight line then it is determined that the cluster is a potential moving obstacle (one or two human legs together, a scan of a typical baby cart, or one of the simulated humans described in Sect. 5) and it is removed from the scan data. With this method small moving obstacles can be erased from the scan data for map building and localization. The drawback is that some small clusters which are part of the environment are removed as well; however, the noisy moving measurements are erased from the scene, improving the resulting map and the localization performance.

1.2 Map building

As many variations of SLAM exist in the field of robot navigation, we aimed to use commonly-available tools and techniques for our comparison.

Map generation was performed by driving the robot slowly through the environment, avoiding frequent turns whenever possible. In our algorithm, we first erased from the laser scans small features, such as human legs, shopping carts, and baby strollers from the raw data, based on the cluster analysis criteria explained in the previous subsection. The laser scan and odometry data from the robot was recorded for this map-building run, and a raster map was built offline by an ICP-based SLAM to correct the trajectory of the robot and align the scans (Borrmann et al. 2008) using 3DTK SLAM software using 2D data (see footnote 2). With the resulting data we built a grid map (Moravec and Elfes 1985; Elfes 1989) with the same coordinate system as the human tracking system.

1.3 Localization technique

To generate map-matching localization estimates for the experiment trials, we replayed laser scan data and odometry data in an offline simulator. Based on this data, our localization algorithm used a particle filter to estimate the robot’s position. The particle filter estimates the robot position with a weighted set of particles. The position of each particle is estimated using wheel encoder data. Particle likelihoods are updated using the previously constructed grid map with a ray tracing approach (Fox 2003; Fox et al. 1999). In our implementation, we used 200 particles for the robot position estimation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Glas, D.F., Morales, Y., Kanda, T. et al. Simultaneous people tracking and robot localization in dynamic social spaces. Auton Robot 39, 43–63 (2015). https://doi.org/10.1007/s10514-015-9426-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-015-9426-3

Keywords

Search

Navigation