Estimation of Subjective Evaluation of HRI Performance Based on Objective Behaviors of Human and Robots
The conventional approach to the evaluation of the performance of human-robot interaction (HRI) is subjective evaluation, such as the application of questionnaires. As such subjective evaluation is time-consuming, an alternative automatic evaluation method based on only objectively observable factors (i.e., human reaction behavior) is required for autonomous learning by robots and for scoring in robot competitions. To this end, we aim to investigate the extent to which subjective evaluation results can be approximated using objective factors. As a case study, we designed and carried out a VR-based robot-competition task in which the robot was required to generate comprehensible and unambiguous natural language expressions and gestures to guide inexpert users in everyday environments. In the competition, both event data and human behavioral data (i.e., interaction histories) were observed and stored. Additionally, to acquire subjective evaluation results, we asked third-parties to evaluate the HRI performance by reviewing the stored interaction histories. From the analysis of the relationship between objective factors and subjective evaluation results, we demonstrate that the subjective evaluation of HRI can indeed be reasonably approximated on the basis of objective factors.
KeywordsHuman-robot interaction Natural language generation RoboCup@Home Virtual reality
- 2.Brščić, D., Kidokoro, H., Suehiro, Y., Kanda, T.: Escaping from children’s abuse of social robots. In: Proceedings of ACM/IEEE International Conference on Human-Robot Interaction, pp. 59–66 (2015)Google Scholar
- 3.Inamura, T., Mizuchi, Y.: Competition design to evaluate cognitive functions in human-robot interaction based on immersive VR. In: Akiyama, H., Obst, O., Sammut, C., Tonidandel, F. (eds.) RoboCup 2017. LNCS (LNAI), vol. 11175, pp. 84–94. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00308-1_7CrossRefGoogle Scholar
- 7.Mizuchi, Y., Inamura, T.: Cloud-based multimodal human-robot interaction simulator utilizing ROS and unity frameworks. In: IEEE/SICE International Symposium on System Integration, pp. 948–955 (2017)Google Scholar
- 10.Striegnitz, K., Denis, A., Gargett, A., Garouf, K., Koller, A., Theune, M.: Report on the second second challenge on generating instructions in virtual environments (GIVE-2.5). In: Proceedings of European Workshop on Natural Language Generation, pp. 270–279 (2011)Google Scholar