Integrating AI Components in a Domestic Robot System


This paper describes the SocRob@Home robot system, consisting of a mobile robot (MBOT) equipped with several sensors and actuators, including a manipulator arm, and several software modules that provide the skills and capability to perform domestic tasks while interacting with humans in a domestic environment. We describe the whole system holistically, explaining how it integrates the contributing modules, and then we focus on the most relevant sub-systems, pointing out the original contributions of our research and development on the system in the last 5 years. The robot system includes metric and semantic mapping, several navigation modes (way-point navigation, person following and multi-sensor obstacle detection and avoidance), vision-based object detection, recognition, servoing and grasping, speech understanding, task planning and task execution. The robot system is mostly activated by speech commands from a human, and these commands, after being interpreted, are executed by the robot sub-systems, coordinated by a task executor. Lessons learned during the development and use of this system, which are useful as guidelines for the development of similar robot systems, are provided. MBOT’s performance is assessed using the task benchmarks scoring system of the European Robotics League competitions on Consumer Service robots.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

  2. 2.

  3. 3.

  4. 4.

  5. 5.

  6. 6.

  7. 7.

  8. 8.

  9. 9.

  10. 10.

  11. 11.

    \(\varGamma\) is set to an arbitrarily small ball around the goal.

  12. 12.


  1. 1.

    Bellotto N, Hanheide M, Dondrup C (2014) BayesTracking: C++ framework for Bayesian filter tracking (UKF, EKF, Particles).

  2. 2.

    Bellotto N, Hu H (2010) Computationally efficient solutions for tracking people with a mobile robot: an experimental evaluation of Bayesian filters. Auton Robots 28(4):425–438

    Article  Google Scholar 

  3. 3.

    Brock O, Khatib O (1999) High-speed navigation using the global dynamic window approach. Proc IEEE Int Conf Robotics Autom 1:341–346 (IEEE)

    Article  Google Scholar 

  4. 4.

    Brzozowska E, Lima O, Ventura R (2019) A generic optimization based cartesian controller for robotic mobile manipulation. In: Proceedings of the IEEE international conference on robotics and automation (ICRA-19). (accepted)

  5. 5.

    Cartucho J, Ventura R, Veloso M (2018) Robust object recognition through symbiotic deep learning in mobile robots. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2336–2341 (IEEE)

  6. 6.

    Cashmore M, Fox M, Long D, Magazzeni D, Ridder B, Carrera A, Palomeras N, Hurtos N, Carreras M (2015) Rosplan: planning in the robot operating system. In: ICAPS, pp 333–341

  7. 7.

    Central Lab Facilities (CITEC) and SocRob project at ISR/IST: bayes\_people\_tracker. (2018).

  8. 8.

    Chaumette F, Hutchinson S (2006) Visual servo control. I. Basic approaches. IEEE Robot Autom Mag 13(4):82–90

    Article  Google Scholar 

  9. 9.

    Chitta S, Sucan I, Cousins S (2012) Moveit![ROS topics]. IEEE Robot Autom Mag 19(1):18–19

    Article  Google Scholar 

  10. 10.

    De Raedt L, Kimmig A (2015) Probabilistic (logic) programming concepts. Mach Learn 100(1):5–47.

    MathSciNet  Article  MATH  Google Scholar 

  11. 11.

    Forlizzi J, DiSalvo C (2006) Service robots in the domestic environment: a study of the roomba vacuum in the home. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human–robot interaction, pp 258–265. ACM

  12. 12.

    Fox D, Burgard W, Thrun S (1997) The dynamic window approach to collision avoidance. IEEE Robot Autom Mag 4(1):23–33

    Article  Google Scholar 

  13. 13.

    Girshick R (2015) Fast R-CNN. In: IEEE Int’l Conf. computer vision (ICCV), pp 1440–1448

  14. 14.

    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conf. computer vision and pattern recognition (CVPR), pp 580–587

  15. 15.

    Gonçalves J, Lima PU (2019) Grasp planning with incomplete knowledge about the object to be grasped. In: Proc. of 19th IEEE Int. Conf. on autonomous robot systems and competitions (ICARSC). IEEE

  16. 16.

    Han J, Jo M, Park S, Kim S (2005) The educational use of home robots for children. In: Robot and human interactive communication, 2005. ROMAN 2005. IEEE International Workshop on, pp 378–383. IEEE

  17. 17.

    Hess W, Kohler D, Rapp H, Andor D (2016) Real-time loop closure in 2D lidar slam. In: 2016 IEEE international conference on robotics and automation (ICRA), pp 1271–1278

  18. 18.

    Hirai K, Hirose M, Haikawa Y, Takenaka T (1998) The development of Honda humanoid robot. Robot Autom 2018, Proc 1998 IEEE Int Conf 2:1321–1326 (IEEE)

    Google Scholar 

  19. 19.

    Katz M, Hoffmann J (2014) Mercury planner: pushing the limits of partial delete relaxation. In: Proceedings of the 8th international planning competition (IPC-2014)

  20. 20.

    Kinarullathil1 M, Martins PH, Azevedo C, Lima O, Lawless G, Lima PU, Custódio L, Ventura R (2018) From user spoken commands to robot task plans: a case study in RoboCup@Home. In: Proc. of language and robotics workshop of IROS2018. IEEE

  21. 21.

    Kraetzschmar G, Basiri M, Lima PU, Ahmad A, Amigoni F, Awaad I, Berghofer J, Bonarini RBA, Dwiputra R, Fontana G, Hegger F, Hochgeschwender N, Iocchi L, Matteucci M, Nardi D, Miraldo P, Schiaffonati V, Schneider S, Bastianelli E, Resende P, Mendes J (2018) European Robotics League for consumer service robots rulebook, 9 October. Available online at:

  22. 22.

    Lima O, Ventura R, Awaad I (2018) Integrating classical planning and real robots in industrial and service robotics domains. PlanRob, ICAPS Workshop

  23. 23.

    Lima PU (2018) A probabilistic approach to benchmarking and performance evaluation of robot systems. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 7231–7236. IEEE

  24. 24.

    Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft COCO: common objects in context. In: European conference on computer vision, pp 740–755. Springer

  25. 25.

    Martins PH, Custódio L, Ventura R (2018) A deep learning approach for understanding natural language commands for mobile service robots. CoRR abs/1807.03053. arXiv:1807.03053

  26. 26.

    Matamoros M, Seib V, Memmesheimer R, Paulus D (2018) Robocup@Home: summarizing achievements in over 11 years of competition. In: Proceedings of 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC 2018), pp 186–191, April, Torres Vedras, Portugal

  27. 27.

    McDermott DM (2000) The 1998 AI planning systems competition. AI Mag 21(2):35

    Google Scholar 

  28. 28.

    Melo FS, Sardinha A, Belo D, Couto M, Faria M, Farias A, Gambôa H, Jesus C, Kinarullathil M, Lima P, Luz L, Mateus A, Melo I, Moreno P, Osório D, Paiva A, Pimentel J, Rodrigues J, Sequeira P, Solera-Ureña R, Vasco M, Veloso M, Ventura R (2018) Project INSIDE: towards autonomous semi-unstructured human–robot social interaction in autism therapy. Artif Intell Med 96:198–216

    Article  Google Scholar 

  29. 29.

    Messias J, Ventura R, Lima P, Sequeira J, Alvito P, Marques C, Carriço P (2014) A robotic platform for edutainment activities in a pediatric hospital. In: Proceedings of the IEEE international conference on autonomous robot systems and competitions (ICARSC), pp 193–198

  30. 30.

    Montemerlo M, Thrun S (2007) Fastslam 2.0. FastSLAM: s scalable method for the simultaneous localization and mapping problem in robotics. Springer, New York, pp 63–90

    Google Scholar 

  31. 31.

    Pangercic D, Pitzer B, Tenorth M, Beetz M (2012) Semantic object maps for robotic housework-representation, acquisition and use. In: Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, pp 4644–4651. IEEE

  32. 32.

    Redmon J (2013–2016) Darknet: open source neural networks in c.

  33. 33.

    Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767

  34. 34.

    Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of Neural Information Processing Systems (NIPS 2015), pp 91–99, Montreal, Canada, December

  35. 35.

    Rusu RB, Blodow N, Marton ZC, Beetz M (2009) Close-range scene segmentation and reconstruction of 3D point cloud maps for mobile manipulation in domestic environments. In: Intelligent robots and systems, 2009. IROS 2009. IEEE/RSJ international conference on, pp 1–6. IEEE

  36. 36.

    Sethian JA (1999) Fast marching methods. SIAM Rev 41(2):199–235

    MathSciNet  Article  Google Scholar 

  37. 37.

    Spaan MT, Veiga TS, Lima PU (2015) Decision-theoretic planning under uncertainty with information rewards for active cooperative perception. Auton Agents Multi-agent Syst 29(6):1157–1185.

    Article  Google Scholar 

  38. 38.

    Stückler J, Holz D, Behnke S (2012) Demonstrating everyday manipulation skills in robocup@home. Robot Autom Mag 19(2):34–42

    Article  Google Scholar 

  39. 39.

    Talebpour Z, Geetha Viswanathan D, Ventura R, Englebienne G, Martinoli A (2016) Incorporating perception uncertainty in human-aware navigation: a comparative study. In: IEEE international symposium on robot and human interactive communication (RO-MAN)

  40. 40.

    Veiga TS, Silva M, Ventura R, Lima PU (2019) An hierarchical approach to active semantic mapping using probabilistic logic and information reward POMDP. In: Proc. of 29th international conference on automated planning and scheduling (ICAPS)

  41. 41.

    Ventura R, Ahmad A (2015) RoboCup 2014: Robot World Cup XVIII, chap. Towards optimal robot navigation in domestic spaces, pp 318–331. LNAI 8992. Springer.

    Google Scholar 

  42. 42.

    Wasik A, Pereira JN, Ventura R, Lima PU, Martinoli A (2016) Graph-based distributed control for adaptive multi-robot patrolling using local formation reconfiguration. In: IEEE/RSJ international conference on intelligent robots and systems (IROS)

Download references

Author information



Corresponding author

Correspondence to Pedro U. Lima.

Additional information

This work was partially supported by ISR/LARSyS Strategic Funding through the FCT project PEst-OE/EEI/LA0009/2013 and by the FCT project HARODE PTDC/EEI-SII/4698/2014.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lima, P.U., Azevedo, C., Brzozowska, E. et al. SocRob@Home. Künstl Intell 33, 343–356 (2019).

Download citation


  • System-level AI for service robots
  • Reasoning with uncertain knowledge
  • Semantic mapping
  • Speech understanding
  • Mobile manipulation
  • Navigation