Skip to main content

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12778))

Included in the following conference series:

Abstract

Domestic service robots are becoming more ubiquitous and can perform various assistive tasks such as fetching items or helping with medicine intake to support humans with impairments of varying severity. However, the development of robots taking care of humans should not only be focused on developing advanced functionalities, but should also be accompanied by the definition of benchmarking protocols enabling the rigorous and reproducible evaluation of robots and their functionalities. Thereby, of particular importance is the assessment of robots’ ability to deal with failures and unexpected events which occur when they interact with humans in real-world scenarios. For example, a person might drop an object during a robot-human hand over due to its weight. However, the systematic investigation of hazardous situations remains challenging as (i) failures are difficult to reproduce; and (ii) possibly impact the health of humans. Therefore, we propose in this paper to employ the concept of scientific robotic competitions as a benchmarking protocol for assessing care robots and to collect datasets of human-robot interactions covering a large variety of failures which are present in real-world domestic environments. We demonstrate the process of defining the benchmarking procedure with the human-to-robot and robot-to-human handover functionalities, and execute a dry-run of the benchmarks while inducing several failure modes such as dropping objects, ignoring the robot, and not releasing objects. A dataset comprising colour and depth images, a wrist force-torque sensor and other internal sensors of the robot was collected during the dry-run. In addition, we discuss the relation between benchmarking protocols and standards that exist or need to be extended with regard to the test procedures required for verifying and validating conformance to standards.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://metricsproject.eu/.

  2. 2.

    https://github.com/HEART-MET/metrics_refbox.

References

  1. Darpa Robotics Challenge. https://archive.darpa.mil/roboticschallenge/. Accessed Jan 2021

  2. European Robotics League. https://www.eu-robotics.net/robotics_league/. Accessed Jan 2021

  3. RoboCup. https://www.robocup.org/. Accessed Jan 2021

  4. Robot Competitions Kick Innovation In Cognitive Systems and Robotics. http://rockinrobotchallenge.eu/. Accessed Jan 2021

  5. Robotics Coordination Action for Europe Two. https://www.eu-robotics.net/eurobotics/about/projects/rockeu2.html. Accessed Jan 2021

  6. World Robot Challenge. https://worldrobotsummit.org/en/wrs2020/challenge/. Accessed Jan 2021

  7. Robotics 2020 Multi-Annual Roadmap for Robotics in Europe. Technical guide, SPARC - Partnership for Robotics in Europe (2016)

    Google Scholar 

  8. Amigoni, F., et al.: Competitions for benchmarking: task and functionality scoring complete performance assessment. IEEE Robot. Autom. Mag. 22(3), 53–61 (2015)

    Google Scholar 

  9. Basiri, M., Piazza, E., Matteucci, M., Lima, P.: Benchmarking functionalities of domestic service robots through scientific competitions. KI - Künstliche Intelligenz 33(4), 357–367 (2019). https://doi.org/10.1007/s13218-019-00619-9

  10. Cavallo, F., et al.: Robotic services acceptance in smart environments with older adults: user satisfaction and acceptability study. J. Med. Internet Res. 20(9) (2018)

    Google Scholar 

  11. Falco, J., et al.: Benchmarking protocols for evaluating grasp strength, grasp cycle time, finger strength, and finger repeatability of robot end-effectors. IEEE Robot. Autom. Lett. 5(2), 644–651 (2020)

    Google Scholar 

  12. Feil-Seifer, D., Skinner, K., Matarić, M.J.: Benchmarks for evaluating socially assistive robotics. Interaction Stud. 8(3), 423–439 (2007)

    Google Scholar 

  13. Hochgeschwender, N., Thoduka, S., Dragone, M., Caleb-Solly, P., Bellamy, D., Cavallo, F.: HEART-MET Evaluation Plan (2020). https://metricsproject.eu/healthcare/. Accessed February 2021

  14. Safety of machinery – General principles for design – Risk assessment and risk reduction. Standard, International Organization for Standardization (2010)

    Google Scholar 

  15. Robots and robotic devices - Safety requirements for personal care robots. Standard, International Organization for Standardization (2014)

    Google Scholar 

  16. Robots and robotic devices – Collaborative robots. Standard, International Organization for Standardization (2016)

    Google Scholar 

  17. Robotics - Performance criteria and related test methods for service robots – Part 1: Locomotion for wheeled robots. Standard, International Organization for Standardization (2016)

    Google Scholar 

  18. Robotics - Performance criteria and related test methods for service robots – Part 2: Navigation. Standard, International Organization for Standardization (2019)

    Google Scholar 

  19. Robotics – Application of ISO 13482 – Part 1: Safety-related test methods. Standard, International Organization for Standardization (2010)

    Google Scholar 

  20. Robotics – Application of ISO 13482 – Part 2: Application guidelines. Standard, International Organization for Standardization (2010)

    Google Scholar 

  21. Kimble, K., et al.: Benchmarking protocols for evaluating small parts robotic assembly systems. IEEE Robot. Autom. Lett. 5(2), 883–889 (2020)

    Google Scholar 

  22. Kyrarini, M., Lygerakis, F., Rajavenkatanarayanan, A., Sevastopoulos, C., Nambiappan, H.R., Chaitanya, K.K., Babu, A.R., Mathew, J., Makedon, F.: A Survey of Robots in Healthcare. Technologies 9(1), 8 (2021)

    Google Scholar 

  23. Lin, C.-C., Liao, H.-Y., Tung, F.-W.: Design guidelines of social-assisted robots for the elderly: a mixed method systematic literature review. In: Stephanidis, C., et al. (eds.) HCII 2020. LNCS, vol. 12425, pp. 90–104. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60128-7_7

    Chapter  Google Scholar 

  24. Morgan, A.S., et al.: Benchmarking cluttered robot pick-and-place manipulation with the box and blocks test. IEEE Robot. Autom. Lett. 5(2), 454–461 (2019)

    Google Scholar 

  25. Rossi, S., et al.: The role of personality factors and empathy in the acceptance and performance of a social robot for psychometric evaluations. Robotics 9(2), 39 (2020)

    Google Scholar 

  26. Sprunk, C., et al.: An experimental protocol for benchmarking robotic indoor navigation. In: Hsieh, M.A., Khatib, O., Kumar, V. (eds.) Experimental Robotics. STAR, vol. 109, pp. 487–504. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-23778-7_32

  27. Tolmeijer, S., et al.: Taxonomy of trust-relevant failures and mitigation strategies. In: Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pp. 3–12 (2020)

    Google Scholar 

  28. Triantafyllou, P., Mnyusiwalla, H., Sotiropoulos, P., Roa, M.A., Russell, D., Deacon, G.: A Benchmarking Framework for Systematic Evaluation of Robotic Pick-and-Place Systems in an Industrial Grocery Setting. In: 2019 International Conference on Robotics and Automation (ICRA). pp. 6692–6698. IEEE (2019)

    Google Scholar 

  29. Weisz, J., Huang, Y., Lier, F., Sethumadhavan, S., Allen, P.: Robobench: towards sustainable robotics system benchmarking. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 3383–3389. IEEE (2016)

    Google Scholar 

Download references

Acknowledgement

This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 871252 (METRICS).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Santosh Thoduka .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Thoduka, S., Hochgeschwender, N. (2021). Benchmarking Robots by Inducing Failures in Competition Scenarios. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. AI, Product and Service. HCII 2021. Lecture Notes in Computer Science(), vol 12778. Springer, Cham. https://doi.org/10.1007/978-3-030-77820-0_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77820-0_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77819-4

  • Online ISBN: 978-3-030-77820-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics