International Journal of Social Robotics

, Volume 2, Issue 2, pp 121–136 | Cite as

A Bank of Unscented Kalman Filters for Multimodal Human Perception with Mobile Service Robots

  • Nicola BellottoEmail author
  • Huosheng Hu


A new generation of mobile service robots could be ready soon to operate in human environments if they can robustly estimate position and identity of surrounding people. Researchers in this field face a number of challenging problems, among which sensor uncertainties and real-time constraints. In this paper, we propose a novel and efficient solution for simultaneous tracking and recognition of people within the observation range of a mobile robot. Multisensor techniques for legs and face detection are fused in a robust probabilistic framework to height, clothes and face recognition algorithms. The system is based on an efficient bank of Unscented Kalman Filters that keeps a multi-hypothesis estimate of the person being tracked, including the case where the latter is unknown to the robot. Several experiments with real mobile robots are presented to validate the proposed approach. They show that our solutions can improve the robot’s perception and recognition of humans, providing a useful contribution for the future application of service robotics.


Robot perception Human tracking and recognition Bayesian estimation Service robotics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arras KO, Mozos OM, Burgard W (2007) Using boosted features for the detection of people in 2d range data. In: Proc of IEEE int conf on robotics and automation (ICRA), Rome, Italy, pp 3402–3407 Google Scholar
  2. 2.
    Asoh H, Vlassis N, Motomura Y, Asano F, Hara I, Hayamizu S, Ito K, Kurita T, Matsui T, Bunschoten R, Kröse B (2001) Jijo-2: An office robot that communicates and learns. IEEE Intell Syst 16(5):46–55 Google Scholar
  3. 3.
    Bar-Shalom Y, Li XR (1995) Multitarget-multisensor tracking: principles and techniques. Y. Bar-Shalom Google Scholar
  4. 4.
    Bar-Shalom Y, Li XR, Kirubarajan T (2001) Estimation with applications to tracking and navigation. Wiley, New York CrossRefGoogle Scholar
  5. 5.
    Bellotto N, Hu H (2007) Multisensor data fusion for joint people tracking and identification with a service robot. In: Proc of IEEE int conf on robotics and biomimetics (ROBIO), Sanya, China, pp 1494–1499 Google Scholar
  6. 6.
    Bellotto N, Hu H (2009) Multisensor-based human detection and tracking for mobile service robots. IEEE Trans Syst Man Cybern, Part B 39(1):167–181 CrossRefGoogle Scholar
  7. 7.
    Bellotto N, Hu H (2010) Computationally efficient solutions for tracking people with a mobile robot: an experimental evaluation of bayesian filters. Auton Robots. doi: 10.1007/s10514-009-9167-2 Google Scholar
  8. 8.
    Bennewitz M, Burgard W, Thrun S (2002) Learning motion patterns of persons for mobile service robots. In: Proc of IEEE int conf on robotics and automation (ICRA), Washington, DC, USA, pp 3601–3606 Google Scholar
  9. 9.
    Bennewitz M, Cielniak G, Burgard W (2003) Utilizing learned motion patterns to robustly track persons. In: Proc of IEEE int w on VS-PETS, France, pp 102–109 Google Scholar
  10. 10.
    Bennewitz M, Burgard W, Cielniak G, Thrun S (2005) Learning motion patterns of people for compliant robot motion. Int J Robot Res 24(1):31–48 CrossRefGoogle Scholar
  11. 11.
    Beveridge R, Bolme D, Teixeira M, Draper B (2003) The CSU face identification evaluation system user’s guide: version 5.0. Computer Science Department, Colorado State University Google Scholar
  12. 12.
    Beymer D, Konolige K (2001) Tracking people from a mobile platform. In: IJCAI workshop on reasoning with uncertainty in robotics, Seattle, WA, USA Google Scholar
  13. 13.
    Blanco J, Burgard W, Sanz R, Fernánez J (2003) Fast face detection for mobile robots by integrating laser range data with vision. In: Proc of the int conf on advanced robotics (ICAR), vol 2, Coimbra, Portugal, pp 953–958 Google Scholar
  14. 14.
    Chakravarty P, Jarvis R (2006) Panoramic vision and laser range finder fusion for multiple person tracking. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), Beijing, China, pp 2949–2954 Google Scholar
  15. 15.
    Cielniak G, Duckett T (2003) Person identification by mobile robots in indoor environments. In Proc of the IEEE int workshop on robotic sensing (ROSE), Örebro, Sweden Google Scholar
  16. 16.
    Cielniak G, Duckett T (2004) People recognition by mobile robots. In: Proc of AILS 2nd joint SAIS/SSLS workshop, Lund, Sweden Google Scholar
  17. 17.
    Comaniciu D, Ramesh V, Meer P (2000) Real-time tracking of non-rigid objects using mean shift. In: Proc of IEEE conf on computer vision and pattern recognition (CVPR), vol 2, SC, USA, pp 142–149 Google Scholar
  18. 18.
    Cunado D, Nixon MS, Carter JN (2003) Automatic extraction and description of human gait models for recognition purposes. Comput Vis Image Underst 90(1):1–41 CrossRefGoogle Scholar
  19. 19.
    Dautenhahn K (1995) Getting to know each other—artificial social intelligence for autonomous robots. Robot Auton Syst 16:333–356 CrossRefGoogle Scholar
  20. 20.
    Fasel I, Fortenberry B, Movellan J (2005) A generative framework for realtime object detection and classification. Comput Vis Image Underst 98:182–210 CrossRefGoogle Scholar
  21. 21.
    Feyrer S, Zell A (2000) Robust real-time pursuit of persons with a mobile robot using multisensor fusion. In: Proc of the 6th int conf on intelligent autonomous systems (IAS), Venice, Italy, pp 710–715 Google Scholar
  22. 22.
    Fong T, Nourbakhsh I, Dautenhahn K (2003) A survey of socially interactive robots. Robot Auton Syst 42(3–4):143–166 zbMATHCrossRefGoogle Scholar
  23. 23.
    Fritsch J, Kleinehagenbrock M, Lang S, Plötz T, Fink GA, Sagerer G (2003) Multi-modal anchoring for human-robot-interaction. Robot Auton Syst 43(2–3):133–147 CrossRefGoogle Scholar
  24. 24.
    Gordon NJ, Maskell S, Kirubarajan T (2002) Efficient particle filters for joint tracking and classification. In: Proc of signal and data processing of small targets (SPIE), FL, USA, pp 439–449 Google Scholar
  25. 25.
    Gorodnichy D (2003) Facial recognition in video. In: Proc of int conf on audio- and video-based biometric person authentication (AVBPA), Guildford, United Kingdom, pp 505–514 Google Scholar
  26. 26.
    Jain AK, Ross A, Prabhakar S (2004) An introduction to biometric recognition. IEEE Trans Circuits Syst Video Technol 14(1):4–20 CrossRefGoogle Scholar
  27. 27.
    Julier SJ, Uhlmann JK (2004) Unscented filtering and nonlinear estimation. Proc IEEE 92(3):401–422 CrossRefGoogle Scholar
  28. 28.
    Li G, Cai X, Li X, Liu Y (2006) An efficient face normalization algorithm based on eyes detection. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), Beijing, China, pp 3843–3848 Google Scholar
  29. 29.
    Lindström M, Eklundh J-O (2001) Detecting and tracking moving objects from a mobile platform using a laser range scanner. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), vol 3, Maui, HI, USA, pp 1364–1369 Google Scholar
  30. 30.
    Liu JNK, Wang M, Feng B (2005) iBotGuard: an internet-based intelligent robot security system using invariant face recognition against intruder. IEEE Trans Syst Man Cybern, Part C 35(1):97–105 CrossRefGoogle Scholar
  31. 31.
    Martin C, Schaffernicht E, Scheidig A, Gross H-M (2005) Sensor fusion using a probabilistic aggregation scheme for people detection and tracking. In: Proc of the 2nd European conference on mobile robots (ECMR), Ancona, Italy, pp 176–181 Google Scholar
  32. 32.
    Minvielle P, Marrs A, Maskell S, Doucet A (2005) Joint target tracking and identification—part II: Shape video computing. In: Proc of the 8th int conf on information fusion, Philadelphia, PA, USA Google Scholar
  33. 33.
    Nakajima C, Pontil M, Heisele B, Poggio T (2003) Full-body person recognition system. Pattern Recogn 36(9):1997–2006 CrossRefGoogle Scholar
  34. 34.
    Nourbakhsh I, Kunz C, Willeke T (2003) The mobot museum robot installations: a five year experiment. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), pp 3636–3641 Google Scholar
  35. 35.
    Pérez P, Vermaak J, Blake A (2004) Data fusion for visual tracking with particles. Proc IEEE 92(3):495–513 CrossRefGoogle Scholar
  36. 36.
    Ristic B, Gordon N, Bessell A (2004) On target classification using kinematic data. Inf Fusion 5(1):15–21 CrossRefGoogle Scholar
  37. 37.
    Santana MC, Suarez OD, Canalis LA, Navarro JL (2008) Face and facial feature detection evaluation. In: Proc of the 3rd int conf on computer vision theory and applications (VISAPP), pp 167–172 Google Scholar
  38. 38.
    Scheutz M, McRaven J, Cserey G (2004) Fast, reliable, adaptive, bimodal people tracking for indoor environments. In: Proc of IEEE/RSJ int conf on intelligent robots and systems (IROS), vol 2, Sendai, Japan, pp 1347–1352 Google Scholar
  39. 39.
    Schulz D, Burgard W, Fox D, Cremers AB (2003) People tracking with mobile robots using sample-based joint probabilistic data association filters. Int J Robot Res 22(2):99–116 CrossRefGoogle Scholar
  40. 40.
    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):72–86 CrossRefGoogle Scholar
  41. 41.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154 CrossRefGoogle Scholar
  42. 42.
    Zajdel W, Zivkovic Z, Krŏse BJA (2005) Keeping track of humans: Have I seen this person before? In: Proc of IEEE int conf on robotics and automation (ICRA), Barcelona, Spain, pp 2093–2098 Google Scholar
  43. 43.
    Zhao W, Chellappa R, Phillips PJ, Rosenfeld A (2003) Face recognition: a literature survey. ACM Comput Surv 35(4):399–458 CrossRefGoogle Scholar
  44. 44.
    Zhou S, Chellappa R (2002) Probabilistic human recognition from video. In: Proc of the 7th European conference on computer vision (ECCV), Springer, London, pp 681–697 Google Scholar

Copyright information

© Springer Science & Business Media BV 2010

Authors and Affiliations

  1. 1.School of Computer ScienceUniversity of LincolnLincolnUK
  2. 2.School of Computer Science and Electronic EngineeringUniversity of EssexColchesterUK

Personalised recommendations