Multimodal sensory fusion for soccer robot self-localization based on long short-term memory recurrent neural network

  • Wenhuan Lu
  • Ju Zhang
  • Xinli Zhao
  • Jianrong Wang
  • Jianwu Dang
Original Research


Self-localization is a fundamental requirement for autonomous mobile robots. With the rapid development in sensor technology, the sensor suites of robot provide multimodal information that naturally ensures perception robustness, multimodal sensory fusion are able to provide a better solution for enhance the capability of self-localization. This paper proposes a multimodal sensory fusion method based on Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) for RoboCup 3D Simulation league. This approach fuses Inertia Navigation System (INS) and vision perceptor information from different sensors at feature level instead of raw data. The experiment results demonstrate that the proposed approach makes an improvement in predictive accuracy and efficiency compared with the standard Extended Kalman Filter (EKF) and the static Particle Filter (PF) methods.


Robot self-localization Multimodal sensory fusion Long short-term memory Recurrent neural network 



The research is supported by part of the National Natural Science Foundation (Surface Project No. 61304250).


  1. Aguirre E, Garcia-Silvente M, Pascual D (2016) A multisensor based approach using supervised learning and particle filtering for people detection and tracking. In: Robot 2015: Second Iberian Robotics Conference, pp 645–657. doi: 10.1007/978-3-319-27149-1_50
  2. Chen SY (2012) Kalman filter for robot vision: a survey. IEEE Trans Indus Electron 59(11):4409–4420. doi: 10.1109/TIE.2011.2162714 CrossRefGoogle Scholar
  3. Deng BZ, Miao H, Gen L et al (2011) Self-localization of omni-directional soccer robot based on multi-sensor. Control Theory Appl 28(12):1821–1824.
  4. Djuric PM, Kotecha JH, Zhang J et al (2003) Particle filtering. Signal Process Mag IEEE 20(5):19–38. doi: 10.1109/MSP.2003.1236770 CrossRefMATHGoogle Scholar
  5. Fox D, Hightower J, Liao L et al (2003) Bayesian filtering for location estimation. Pervasive Comput IEEE 2(3):24–33. doi: 10.1109/MPRV.2003.1228524 CrossRefGoogle Scholar
  6. Gers FA, Eck D, Schmidhuber J (2001) Applying LSTM to time series predictable through time-window approaches. In: International Conference on Artificial Neural Networks, pp 669-676. doi: 10.1007/3-540-44668-0_93
  7. Gers FA, Schraudolph NN, Schmidhuber J et al (2003) Learning precise timing with LSTM recurrent networks. J Mach Learn Res 3(1):115–143. doi: 10.1162/153244303768966139 MathSciNetMATHGoogle Scholar
  8. Gouaillier D, Hugel V, Blazevic P et al (2009) Mechatronic design of NAO humanoid. In: IEEE International Conference on Robotics and Automation, pp 769–774. doi: 10.1109/ROBOT.2009.5152516
  9. Hashemi E, Jadid MG, Lashgarian M et al (2012) Particle filter based localization of the Nao biped robots. In: 44th Southeastern Symposium on System Theory (SSST), pp 168–173. doi: 10.1109/SSST.2012.6195128
  10. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. doi: 10.1162/neco.1997.9.8.1735 CrossRefGoogle Scholar
  11. Kam M, Zhu X, Kalata P (1997) Sensor fusion for mobile robot navigation. Proc IEEE 85(1):108–119. doi: 10.1109/JPROC.1997.554212 CrossRefGoogle Scholar
  12. Kim J, Kim Y, Kim S (2008) An accurate localization for mobile robot using extended Kalman filter and sensor fusion. In: 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp 2928–2933. doi: 10.1109/IJCNN.2008.4634210
  13. Kogler M, Obst O (2003) Simulation league: the next generation. In: RoboCup 2003: Robot Soccer World Cup VII, pp 458–469. doi: 10.1007/978-3-540-25940-4_40
  14. Koval V, Adamiv O, Kapura V (2007) The local area map building for mobile robot navigation using ultrasound and infrared sensors. In: IEEE Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, pp 454–459. doi: 10.1109/IDAACS.2007.4488459
  15. Leonard JJ, Durrantwhyte HF (1991) Mobile robot localization by tracking geometric beacons. IEEE Trans Robot Autom 7(3):376–382. doi: 10.1109/70.88147 CrossRefGoogle Scholar
  16. Liang Z, Hao Y (2014) Adaptive particle filter for self-localization of RoboCup 3D soccer robots. Int J Robot Autom 29(3): doi: 10.2316/Journal.206.2014.3.206-3969
  17. Luo RC, Chang CC (2012) Multisensor fusion and integration: a review on approaches and its applications in mechatronics. IEEE Trans Ind Inf 8(1):49–60. doi: 10.1109/TII.2011.2173942 CrossRefGoogle Scholar
  18. Rodriguez FSA, Fremont V, Bonnifait P (2009) An experiment of a 3D real-time robust visual odometry for intelligent vehicles. In: International IEEE Conference on Intelligent Transportation Systems, pp 1–6. doi: 10.1109/ITSC.2009.5309615
  19. Smith D, Singh S (2006) Approaches to multisensor data fusion in target tracking: a survey. IEEE Trans Knowl Data Eng 18(12):1696–1710. doi: 10.1109/TKDE.2006.183 CrossRefGoogle Scholar
  20. Stoecker J, Visser U (2012) RoboViz: programmable visualization for simulated soccer. In: RoboCup 2011: Robot Soccer World Cup XV, pp 282–293. doi: 10.1007/978-3-642-32060-6_24
  21. Thrun S (2002) Probabilistic robotics. Commun Acm 45(3):52–57. doi: 10.1145/504729.504754 CrossRefGoogle Scholar
  22. Titterton D (2004) Strapdown inertial navigation technology. Aerosp Electron Syst Mag IEEE 20(7):33–34. doi: 10.1049/PBRA017E CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Wenhuan Lu
    • 1
  • Ju Zhang
    • 2
  • Xinli Zhao
    • 1
  • Jianrong Wang
    • 2
  • Jianwu Dang
    • 2
  1. 1.School of Computer SoftwareTianjin UniversityTianjinChina
  2. 2.School of Computer Science and TechnologyTianjin UniversityTianjinChina

Personalised recommendations