Data fusion in ubiquitous networked robot systems for urban services

  • Luis Merino
  • Andrew Gilbert
  • Jesús Capitán
  • Richard Bowden
  • John Illingworth
  • Aníbal Ollero


There is a clear trend in the use of robots to accomplish services that can help humans. In this paper, robots acting in urban environments are considered for the task of person guiding. Nowadays, it is common to have ubiquitous sensors integrated within the buildings, such as camera networks, and wireless communications like 3G or WiFi. Such infrastructure can be directly used by robotic platforms. The paper shows how combining the information from the robots and the sensors allows tracking failures to be overcome, by being more robust under occlusion, clutter, and lighting changes. The paper describes the algorithms for tracking with a set of fixed surveillance cameras and the algorithms for position tracking using the signal strength received by a wireless sensor network (WSN). Moreover, an algorithm to obtain estimations on the positions of people from cameras on board robots is described. The estimate from all these sources are then combined using a decentralized data fusion algorithm to provide an increase in performance. This scheme is scalable and can handle communication latencies and failures. We present results of the system operating in real time on a large outdoor environment, including 22 nonoverlapping cameras, WSN, and several robots.


Ubiquitous networked robots Decentralized data fusion Service and social robotics Person tracking 



This work is partially supported by URUS, Ubiquitous networking Robotics in Urban Settings, funded by the European Commission (EC) under FP6 with contract number FP6-EU-IST-045062. In addition, the authors would like to thank to the rest of the partners of the URUS project for their help and support. Luis Merino is also funded by the EC through the project FROG (FP7-288235). Jesus Capitan is also funded by Fundação para a Ciência e a Tecnologia (ISR/IST pluriannual funding) through the PIDDAC Program funds and projects PEst-OE/EEI/LA0009/2011 and CMU-PT/SIA/0023/2009.


  1. 1.
    Bailey T, Durrant-Whyte H (2007) Decentralised data fusion with delayed states for consistent inference in mobile ad hoc networks. Technical report, Australian Centre for Field Robotics, University of SydneyGoogle Scholar
  2. 2.
    Bradski GR (1998) Computer vision face tracking as a component of perceptual user interface. In: Proc. of workshop on applications of computer vision, pp 214–219Google Scholar
  3. 3.
    Caballero F, Merino L, Gil P, Maza I, Ollero A (2008) A probabilistic framework for entire wsn localization using a mobile robot. J Robot Auton Syst 56(10):798–806CrossRefGoogle Scholar
  4. 4.
    Cai Q, Aggarwal JF (1998) Automatic tracking of human motion in indoor scenes across multiple synchronized video streams. In: Proc. of IEEE international conference on computer vision (ICCV’98)Google Scholar
  5. 5.
    Cánovas JP, LeBlanc K, Saffiotti A (2004) Robust multi-robot object localization using fuzzy logic. In: Proc. of the international robocup symposiumGoogle Scholar
  6. 6.
    Capitán J, Merino L, Caballero F, Ollero A (2009) Delayed-state information filter for cooperative decentralized tracking. In: Proceedings of the international conference on robotics and automation, ICRAGoogle Scholar
  7. 7.
    Chang TH, Gong S, Ong E (2000) Tracking multiple people under occlusion using multiple cameras. In: Proc. of BMVA British machine vision conference (BMVC’00), pp 566–575Google Scholar
  8. 8.
    Dick A, Brooks M (2004) A stochastic approach to tracking objects across multiple cameras. In: Proc. of Australian conference on artificial intelligence, pp 160–170Google Scholar
  9. 9.
    Ellis TJ, Makris D, Black JK (2003) Learning a multi-camera topology. In: Proc. of joint IEEE workshop on visual surveillance and performance evaluation of tracking and surveillance (VS-PETS), pp 165–171Google Scholar
  10. 10.
    Ess A, Leibe B, Schindler K, Van Gool L (2008) A mobile vision system for robust multi-person tracking. In: IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  11. 11.
    Farenzena M, Bazzani L, Perina M, Cristani A, Murino V (2010) Person re-identification by symmetry-driven accumulation of local features. In: Proc. of IEEE international conference on computer vision and pattern recognition (CVPR’10)Google Scholar
  12. 12.
    Ferris B, Hohnel D, Fox D (2006) Gaussian processes for signal strength-based location estimation. In: Proc. of robotics science and systemsGoogle Scholar
  13. 13.
    Fintrop S, Königs A, Hoeller F, Schulz D (2009) Visual person tracking using a cognitive observation model. In: International conference on robotics and automation (ICRA), workshop in people detection and trackingGoogle Scholar
  14. 14.
    Gilbert A, Bowden R (2008) Incremental, scalable tracking of objects inter camera. Comput Vis Image Underst (CVIU) 3:43–58CrossRefGoogle Scholar
  15. 15.
    Gilbert A, Capitán J, Bowden R, Merino L (2009) Accurate fusion of robot, camera and wireless sensors for surveillance applications. In: Proc. ninth IEEE international workshop on visual surveillance (ICCV09), Kyoto, JapanGoogle Scholar
  16. 16.
    Gilbert A, Illingworth J, Bowden R (2008) Scale invariant action recognition using compound features mined from dense spatiotemporal corners. In: Proc. of European conference on computer vision (ECCV’08), vol I, pp 222–233Google Scholar
  17. 17.
    Gray D, Tao H (2008) Viewpoint invariant pedestrain recognition with an ensemble of localized features. In: Proc. of European conference on computer vision (ECCV’08), pp 262–275Google Scholar
  18. 18.
    Grime S, Durrant-Whyte HF (1994) Data fusion in decentralized sensor networks. Control Eng Pract 2(5):849–863CrossRefGoogle Scholar
  19. 19.
    Grocholsky B, Makarenko A, Kaupp T, Durrant-Whyte HF (2003) Scalable control of decentralised sensor platforms. In: Lecture notes in computer science, vol 2634. SpringerGoogle Scholar
  20. 20.
    Gustafsson F, Gunnarsson F (2005) Mobile positioning using wireless networks: possibilities and fundamental limitations based on available wireless network measurements IEEE Signal Process Mag 22 (4)41–53CrossRefGoogle Scholar
  21. 21.
    Hollinger G, Djugash J, Singh S (2008) Tracking a moving target in cluttered environments with ranging radios. In: IEEE int conf robot autom, pp 1430–1435. doi: 10.1109/ROBOT.2008.4543403
  22. 22.
    Huang T, Russell S (1997) Object identification in a Bayesian context. In: Proc. of international joint conference on artificial intelligence (IJCAI-97), pp 1276–1283Google Scholar
  23. 23.
    Javed O, Shafique K, Rasheed Z, Shah M (2008) Modeling inter-camera space-time and appearance relationships for tracking across nonoverlapping views. Comput Vis Image Underst 109(2):146–162CrossRefGoogle Scholar
  24. 24.
    Julier SJ, Uhlmann JK (1997) A non-divergent estimation algorithm in the presence of unknown correlations. Proc Am Control Conf 4:2369–2373Google Scholar
  25. 25.
    KaewTrakulPong P, Bowden R (2003) A real-time adaptive visual surveillance system for tracking low resolution colour targets in dynamically changing scenes. J Image Vis Comput 21(10):913–929CrossRefGoogle Scholar
  26. 26.
    KaewTrakulPong P, Bowden R (2005) Towards automated wide area visual surveillance: tracking objects between spatially separated, uncalibrated views. IEE Proc Vis Image Signal Process 152(10):213–224Google Scholar
  27. 27.
    Kelly P, Katkere A, Kuramura D, Moezzi S, Chatterjee S (1995) An architecture for multiple perspective interactive video. In: Proc. of the 3rd ACE international conference on multimedia, pp 201–212Google Scholar
  28. 28.
    Kettnaker V, Zabih R (1999) Bayesian multi-camera surveillance. In: Proc. of international conference on computer vision and pattern recognition, pp 253–259Google Scholar
  29. 29.
    Kusý B, Ledeczi A, Koutsoukos X (2007) Tracking mobile nodes using RF Doppler shifts. In: Proceedings of SenSys, pp 29–42Google Scholar
  30. 30.
    Kusý B, Sallai J, Balogh G, Ledeczi A, Protopopescu V, Tolliver J, DeNap F, Parang M (2007) Radio intereferometric tracking mobile wireless nodes. In: Proceedings of MobySys, pp 139–151Google Scholar
  31. 31.
    Makarenko A, Brooks A, Williams S, Durrant-Whyte H, Grocholsky B (2004) A decentralized architecture for active sensor networks. In: Proceedings IEEE international conference on robotics and automation, ICRA, vol 2, pp 1097–1102Google Scholar
  32. 32.
    Martinez-Mozos O, Kurazume R, Hasegawa T (2010) Multi-part people detection using 2D range data. Int J Soc Rob 2:31–40CrossRefGoogle Scholar
  33. 33.
    Morariu VI, Camps OI (2006) Modeling correspondences for multi-camera tracking using nonlinear manifold learning and target dynamics. In: Proc. of IEEE international conference on computer vision and pattern recognition (CVPR’06), vol I, pp 545–552Google Scholar
  34. 34.
    Morelli C, Nicoli M, Rampa V, Spagnolini U, Alippi C (2006) Particle filters for RSS-based localization in wireless sensor networks: an experimental study. In: International conference on acoustics, speech and signal processing, vol 4, p IVGoogle Scholar
  35. 35.
    Navarro-Serment LE, Mertz C, Hebert M (2009) Pedestrian detection and tracking using three-dimensional ladar data. In: Proc. of the 7th int. conf. on field and service roboticsGoogle Scholar
  36. 36.
    Nettleton E, Durrant-Whyte H, Sukkarieh S (2003) A robust architecture for decentralised data fusion. In: Proc. of the international conference on advanced robotics (ICAR)Google Scholar
  37. 37.
    Norlund PJ, Gustafsson F, Gunnarsson F (2002) Particle filters for positioning in wireless networks. In: Proceedings of EUSIPCOGoogle Scholar
  38. 38.
    Olfati-Saber R, Fax J, Murray R (2007) Consensus and cooperation in networked multi-agent systems. Proc IEEE 95(1):215–233CrossRefGoogle Scholar
  39. 39.
    Prosser B, Gong S, Xiang T (2008) Multi-camera matching using bi-directional cumulative brightness transfer functions. In: BMVC’08, pp 164.1–64.10Google Scholar
  40. 40.
    Ramadurai V, Sichitiu ML (2003) Localization in wireless sensor networks: a probabilistic approach. In: Proceedings of the 2003 international conference on wireless networks (ICWN 2003), Las Vegas, NV, pp 275–281Google Scholar
  41. 41.
    Ren W, Beard R, Atkins E (2007) Information consensus in multivehicle cooperative control. IEEE Control Syst 27(2):71–82CrossRefGoogle Scholar
  42. 42.
    Rosencrantz M, Gordon G, Thrun S (2003) Decentralized sensor fusion with distributed particle filters. In: Proc. conf. uncertainty in artificial intelligenceGoogle Scholar
  43. 43.
    Sanfeliu A, Andrade-Cetto J, Barbosa M, Bowden R, Capitán J, Corominas A, Gilbert A, Illingworth J, Merino L, Mirats JM, Moreno P, Ollero A, Sequeira J, Spaan MTJ (2010) Decentralized sensor fusion for ubiquitous networking robotics in urban areas. Sensors 10(3):2274–2314CrossRefGoogle Scholar
  44. 44.
    Seshadri V, Zaruba GV, Huber M (2005) A Bayesian sampling approach to in-door localization of wireless devices using received signal strength indication. In: International conference on pervasive computing and communications, pp 75–84Google Scholar
  45. 45.
    Stone LD, Corwin TL, Barlow CA (1999) Bayesian multiple target tracking. Artech House, Inc., NorwoodGoogle Scholar
  46. 46.
    Sukkarieh S, Nettleton E, Kim J-H, Ridley M, Goktogan A, Durrant-Whyte H (2003) The ANSER project: data fusion across multiple uninhabited air vehicles. Int J Rob Res 22(7–8):505–539CrossRefGoogle Scholar
  47. 47.
    Uhlmann JK (2003) Covariance consistency methods for fault-tolerant distributed data fusion. Inf Fusion 4(3):201–215CrossRefGoogle Scholar
  48. 48.
    Ukita N (2007) Probabilistic-topological calibration of widely distributed camera networks. Mach Vis Appl 18(3):249–260zbMATHCrossRefGoogle Scholar
  49. 49.
    Viola P, Jones M (2004) Robust real-time face detection. Int J Comput Vis 57:137–154CrossRefGoogle Scholar

Copyright information

© Institut Mines-Télécom and Springer-Verlag 2012

Authors and Affiliations

  • Luis Merino
    • 1
  • Andrew Gilbert
    • 2
  • Jesús Capitán
    • 3
  • Richard Bowden
    • 2
  • John Illingworth
    • 2
  • Aníbal Ollero
    • 4
    • 5
  1. 1.School of EngineeringPablo de Olavide UniversitySevilleSpain
  2. 2.Centre for Vision Speech and Signal ProcessingUniversity of SurreyGuildfordUK
  3. 3.Institute for Systems and RoboticsInstituto Superior TecnicoLisbonPortugal
  4. 4.School of EngineeringUniversity of SevilleSevilleSpain
  5. 5.Centre for Advanced Aerospace TechnologyParque Tecnológico y Aeronáutico de AndalucíaLa RinconadaSpain

Personalised recommendations