Skip to main content

Advertisement

SpringerLink
Log in

A Bank of Unscented Kalman Filters for Multimodal Human Perception with Mobile Service Robots

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

A new generation of mobile service robots could be ready soon to operate in human environments if they can robustly estimate position and identity of surrounding people. Researchers in this field face a number of challenging problems, among which sensor uncertainties and real-time constraints. In this paper, we propose a novel and efficient solution for simultaneous tracking and recognition of people within the observation range of a mobile robot. Multisensor techniques for legs and face detection are fused in a robust probabilistic framework to height, clothes and face recognition algorithms. The system is based on an efficient bank of Unscented Kalman Filters that keeps a multi-hypothesis estimate of the person being tracked, including the case where the latter is unknown to the robot. Several experiments with real mobile robots are presented to validate the proposed approach. They show that our solutions can improve the robot’s perception and recognition of humans, providing a useful contribution for the future application of service robotics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Arras KO, Mozos OM, Burgard W (2007) Using boosted features for the detection of people in 2d range data. In: Proc of IEEE int conf on robotics and automation (ICRA), Rome, Italy, pp 3402–3407

  2. Asoh H, Vlassis N, Motomura Y, Asano F, Hara I, Hayamizu S, Ito K, Kurita T, Matsui T, Bunschoten R, Kröse B (2001) Jijo-2: An office robot that communicates and learns. IEEE Intell Syst 16(5):46–55

    Google Scholar 

  3. Bar-Shalom Y, Li XR (1995) Multitarget-multisensor tracking: principles and techniques. Y. Bar-Shalom

  4. Bar-Shalom Y, Li XR, Kirubarajan T (2001) Estimation with applications to tracking and navigation. Wiley, New York

    Book  Google Scholar 

  5. Bellotto N, Hu H (2007) Multisensor data fusion for joint people tracking and identification with a service robot. In: Proc of IEEE int conf on robotics and biomimetics (ROBIO), Sanya, China, pp 1494–1499

  6. Bellotto N, Hu H (2009) Multisensor-based human detection and tracking for mobile service robots. IEEE Trans Syst Man Cybern, Part B 39(1):167–181

    Article  Google Scholar 

  7. Bellotto N, Hu H (2010) Computationally efficient solutions for tracking people with a mobile robot: an experimental evaluation of bayesian filters. Auton Robots. doi:10.1007/s10514-009-9167-2

    Google Scholar 

  8. Bennewitz M, Burgard W, Thrun S (2002) Learning motion patterns of persons for mobile service robots. In: Proc of IEEE int conf on robotics and automation (ICRA), Washington, DC, USA, pp 3601–3606

  9. Bennewitz M, Cielniak G, Burgard W (2003) Utilizing learned motion patterns to robustly track persons. In: Proc of IEEE int w on VS-PETS, France, pp 102–109

  10. Bennewitz M, Burgard W, Cielniak G, Thrun S (2005) Learning motion patterns of people for compliant robot motion. Int J Robot Res 24(1):31–48

    Article  Google Scholar 

  11. Beveridge R, Bolme D, Teixeira M, Draper B (2003) The CSU face identification evaluation system user’s guide: version 5.0. Computer Science Department, Colorado State University

  12. Beymer D, Konolige K (2001) Tracking people from a mobile platform. In: IJCAI workshop on reasoning with uncertainty in robotics, Seattle, WA, USA

  13. Blanco J, Burgard W, Sanz R, Fernánez J (2003) Fast face detection for mobile robots by integrating laser range data with vision. In: Proc of the int conf on advanced robotics (ICAR), vol 2, Coimbra, Portugal, pp 953–958

  14. Chakravarty P, Jarvis R (2006) Panoramic vision and laser range finder fusion for multiple person tracking. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), Beijing, China, pp 2949–2954

  15. Cielniak G, Duckett T (2003) Person identification by mobile robots in indoor environments. In Proc of the IEEE int workshop on robotic sensing (ROSE), Örebro, Sweden

  16. Cielniak G, Duckett T (2004) People recognition by mobile robots. In: Proc of AILS 2nd joint SAIS/SSLS workshop, Lund, Sweden

  17. Comaniciu D, Ramesh V, Meer P (2000) Real-time tracking of non-rigid objects using mean shift. In: Proc of IEEE conf on computer vision and pattern recognition (CVPR), vol 2, SC, USA, pp 142–149

  18. Cunado D, Nixon MS, Carter JN (2003) Automatic extraction and description of human gait models for recognition purposes. Comput Vis Image Underst 90(1):1–41

    Article  Google Scholar 

  19. Dautenhahn K (1995) Getting to know each other—artificial social intelligence for autonomous robots. Robot Auton Syst 16:333–356

    Article  Google Scholar 

  20. Fasel I, Fortenberry B, Movellan J (2005) A generative framework for realtime object detection and classification. Comput Vis Image Underst 98:182–210

    Article  Google Scholar 

  21. Feyrer S, Zell A (2000) Robust real-time pursuit of persons with a mobile robot using multisensor fusion. In: Proc of the 6th int conf on intelligent autonomous systems (IAS), Venice, Italy, pp 710–715

  22. Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3–4):143–166

    Article  MATH  Google Scholar 

  23. Fritsch J, Kleinehagenbrock M, Lang S, Plötz T, Fink GA, Sagerer G (2003) Multi-modal anchoring for human-robot-interaction. Robot Auton Syst 43(2–3):133–147

    Article  Google Scholar 

  24. Gordon NJ, Maskell S, Kirubarajan T (2002) Efficient particle filters for joint tracking and classification. In: Proc of signal and data processing of small targets (SPIE), FL, USA, pp 439–449

  25. Gorodnichy D (2003) Facial recognition in video. In: Proc of int conf on audio- and video-based biometric person authentication (AVBPA), Guildford, United Kingdom, pp 505–514

  26. Jain AK, Ross A, Prabhakar S (2004) An introduction to biometric recognition. IEEE Trans Circuits Syst Video Technol 14(1):4–20

    Article  Google Scholar 

  27. Julier SJ, Uhlmann JK (2004) Unscented filtering and nonlinear estimation. Proc IEEE 92(3):401–422

    Article  Google Scholar 

  28. Li G, Cai X, Li X, Liu Y (2006) An efficient face normalization algorithm based on eyes detection. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), Beijing, China, pp 3843–3848

  29. Lindström M, Eklundh J-O (2001) Detecting and tracking moving objects from a mobile platform using a laser range scanner. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), vol 3, Maui, HI, USA, pp 1364–1369

  30. Liu JNK, Wang M, Feng B (2005) iBotGuard: an internet-based intelligent robot security system using invariant face recognition against intruder. IEEE Trans Syst Man Cybern, Part C 35(1):97–105

    Article  Google Scholar 

  31. Martin C, Schaffernicht E, Scheidig A, Gross H-M (2005) Sensor fusion using a probabilistic aggregation scheme for people detection and tracking. In: Proc of the 2nd European conference on mobile robots (ECMR), Ancona, Italy, pp 176–181

  32. Minvielle P, Marrs A, Maskell S, Doucet A (2005) Joint target tracking and identification—part II: Shape video computing. In: Proc of the 8th int conf on information fusion, Philadelphia, PA, USA

  33. Nakajima C, Pontil M, Heisele B, Poggio T (2003) Full-body person recognition system. Pattern Recogn 36(9):1997–2006

    Article  Google Scholar 

  34. Nourbakhsh I, Kunz C, Willeke T (2003) The mobot museum robot installations: a five year experiment. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), pp 3636–3641

  35. Pérez P, Vermaak J, Blake A (2004) Data fusion for visual tracking with particles. Proc IEEE 92(3):495–513

    Article  Google Scholar 

  36. Ristic B, Gordon N, Bessell A (2004) On target classification using kinematic data. Inf Fusion 5(1):15–21

    Article  Google Scholar 

  37. Santana MC, Suarez OD, Canalis LA, Navarro JL (2008) Face and facial feature detection evaluation. In: Proc of the 3rd int conf on computer vision theory and applications (VISAPP), pp 167–172

  38. Scheutz M, McRaven J, Cserey G (2004) Fast, reliable, adaptive, bimodal people tracking for indoor environments. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), vol 2, Sendai, Japan, pp 1347–1352

  39. Schulz D, Burgard W, Fox D, Cremers AB (2003) People tracking with mobile robots using sample-based joint probabilistic data association filters. Int J Robot Res 22(2):99–116

    Article  Google Scholar 

  40. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):72–86

    Article  Google Scholar 

  41. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  42. Zajdel W, Zivkovic Z, Krŏse BJA (2005) Keeping track of humans: Have I seen this person before? In: Proc of IEEE int conf on robotics and automation (ICRA), Barcelona, Spain, pp 2093–2098

  43. Zhao W, Chellappa R, Phillips PJ, Rosenfeld A (2003) Face recognition: a literature survey. ACM Comput Surv 35(4):399–458

    Article  Google Scholar 

  44. Zhou S, Chellappa R (2002) Probabilistic human recognition from video. In: Proc of the 7th European conference on computer vision (ECCV), Springer, London, pp 681–697

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Lincoln, Lincoln, LN6 7TS, UK

    Nicola Bellotto

  2. School of Computer Science and Electronic Engineering, University of Essex, Colchester, CO4 3SQ, UK

    Huosheng Hu

Authors
  1. Nicola Bellotto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huosheng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicola Bellotto.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bellotto, N., Hu, H. A Bank of Unscented Kalman Filters for Multimodal Human Perception with Mobile Service Robots. Int J of Soc Robotics 2, 121–136 (2010). https://doi.org/10.1007/s12369-010-0047-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-010-0047-x

Keywords

Search

Navigation