Skip to main content
SpringerLink
Log in

Human engagement intention intensity recognition method based on two states fusion fuzzy inference system

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Natural human–robot interaction requires social robots to have human-like perception of engagement intention. In the multi-person interaction scenario, it’s a vital task for social robots to rationally select the main interaction object. Existing studies mostly focus on analyzing whether a person has engagement intention before interaction. However, this qualitative analysis of engagement intention is only applicable in single-person interaction scenarios. When multiple people have the intention to engage with the robot, the robot needs to quantitatively analyze the engagement intention intensity (EII) of all people to make a reasonable interaction decision. In addition, for EII recognition, it is an ideal state that social robots can imitate human social thinking as much as possible. For these purposes, a method that can efficiently recognize the EII by fusing transient features and temporal features is proposed. First, the 3D pose extractor is used to extract the 3D skeleton information which can calculate the transient features including linear distance and body orientation. Second, an improved ConvLSTM network is proposed to effectively identify pedestrian motion states which can reflect temporal information. Finally, based on the proposed two states fusion fuzzy inference system (TSFFIS), the EII can be judged by the three features which are linear distance, body orientation and motion states. Comparative experiments show that our method can effectively identify the EII of different pedestrians relative to the robot. Compared with existing methods, the EII recognition method based on TSFFIS has better performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Salichs MA, Castro-González Á, Salichs E, Fernández-Rodicio E, Maroto-Gómez M et al (2020) Mini: a new social robot for the elderly. Int J Soc Robot 12(6):1231–1249. https://doi.org/10.1007/s12369-020-00687-0

    Article  Google Scholar 

  2. Chen H, Park HW, Breazeal C (2020) Teaching and learning with children: impact of reciprocal peer learning with a social robot on children’s learning and emotive engagement. Comput Educ 150:103836. https://doi.org/10.1016/j.compedu.2020.103836

    Article  Google Scholar 

  3. Lambert A, Norouzi N, Bruder G, Welch G (2020) A systematic review of ten years of research on human interaction with social robots. Int J Hum Comput Interact 36(19):1804–1817. https://doi.org/10.1080/10447318.2020.1801172

    Article  Google Scholar 

  4. Xue Y, Wang F, Tian H, Zhao M, Li J, Pan H, Dong Y. (2021) Proactive interaction framework for intelligent social receptionist robots. In: 2021 IEEE international conference on robotics and automation (ICRA) (pp. 3403–3409). IEEE. https://doi.org/10.1109/ICRA48506.2021.9562115

  5. Sirithunge C, Jayasekara ABP, Chandima DP (2019) Proactive robots with the perception of nonverbal human behavior: a review. IEEE Access 7:77308–77327. https://doi.org/10.1109/ACCESS.2019.2921986

    Article  Google Scholar 

  6. Tasaki T, Matsumoto S, Ohba H, Yamamoto S, Toda M et al (2006) Dynamic communication of humanoid robot with multiple people based on interaction distance. Inf Media Technol 1(1):285–295. https://doi.org/10.11185/imt.1.285

    Article  Google Scholar 

  7. Vaufreydaz D, Johal W, Combe C (2016) Starting engagement detection towards a companion robot using multimodal features. Robot Auton Syst 75:4–16. https://doi.org/10.1016/j.robot.2015.01.004

    Article  Google Scholar 

  8. Truong XT, Ngo TD (2019) Social interactive intention prediction and categorization. In: ICRA 2019 Workshop on MoRobAE-mobile robot assistants for the elderly, Montreal Canada, May 20–24

  9. Ozaki Y, Ishihara T, Matsumura N, Nunobiki T, Yamada T (2018) Decision-making prediction for human-robot engagement between pedestrian and robot receptionist. In: 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN) (pp. 208–215). IEEE. https://doi.org/10.1109/ROMAN.2018.8525814

  10. Bi J, He M, Luo M, Hu F. (2021) Interactive intention prediction model for humanoid robot based on visual features. In: 2021 2nd international conference on control, robotics and intelligent system (pp. 36–41). https://doi.org/10.1145/3483845.3483852

  11. Ren S, Jin G, Liu K, Sun Y, Liang J, Jiang S, Wang J (2019) Research on interactive intent recognition based on facial expression and line of sight direction. In: International conference on advanced data mining and applications (pp. 431–443). Springer, Cham. https://doi.org/10.1007/978-3-030-35231-8_31

  12. Kobayashi Y, Gyoda M, Tabata T, Kuno Y, Yamazaki K et al. (2011) A considerate care robot able to serve in multi-party settings. In: 2011 RO-MAN (pp. 27–32). IEEE. https://doi.org/10.1109/ROMAN.2011.6005286

  13. Whitehill J, Serpell Z, Lin YC, Foster A, Movellan JR (2014) The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans Affect Comput 5(1):86–98. https://doi.org/10.1109/TAFFC.2014.2316163

    Article  Google Scholar 

  14. Sidiropoulos GK, Papakostas GA, Lytridis C, Bazinas C, Kaburlasos VG, et al (2020) Measuring engagement level in child-robot interaction using machine learning based data analysis. In: 2020 International conference on data analytics for business and industry: way towards a sustainable economy (pp. 1–5). IEEE. https://doi.org/10.1109/ICDABI51230.2020.9325676

  15. Li L, Xu Q, Tan YK. (2012) Attention-based addressee selection for service and social robots to interact with multiple persons. In: Proceedings of the Workshop at SIGGRAPH Asia (pp. 131–136). https://doi.org/10.1145/2425296.2425319

  16. Mazhar O, Ramdani S, Navarro B, Passama R, Cherubini A (2018) Towards real-time physical human-robot interaction using skeleton information and hand gestures. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1–6). IEEE. https://doi.org/10.1109/IROS.2018.8594385

  17. Abdelrahman AA, Strazdas D, Khalifa A, Hintz J, Hempel T et al (2022) Multimodal engagement prediction in multiperson human-robot interaction. IEEE Access 10:61980–61991. https://doi.org/10.1109/ACCESS.2022.3182469

    Article  Google Scholar 

  18. Foster ME, Gaschler A, Giuliani M (2017) Automatically classifying user engagement for dynamic multi-party human-robot interaction. Int J Soc Robot 9(5):659–674. https://doi.org/10.1007/s12369-017-0414-y

    Article  Google Scholar 

  19. Hall ET, Birdwhistell RL, Bock B, Bohannan P, Diebold AR Jr et al (1968) Proxemics [and comments and replies]. Curr Anthropol 9(2/3):83–108. https://doi.org/10.1086/200975

    Article  Google Scholar 

  20. Heenan B, Greenberg S, Aghel-Manesh S, Sharlin E (2014) Designing social greetings in human robot interaction. In: Proceedings of the 2014 conference on Designing interactive systems (pp. 855–864). https://doi.org/10.1145/2598510.2598513

  21. Michalowski MP, Sabanovic S, Simmons R. (2006) A spatial model of engagement for a social robot. In: 9th IEEE International Workshop on Advanced Motion Control, 2006. (pp. 762–767). IEEE. https://doi.org/10.1109/AMC.2006.1631755

  22. Feil-Seifer D, Matarić MJ (2012) Distance-based computational models for facilitating robot interaction with children. J Hum Robot Interact 1(1):55–77. https://doi.org/10.5898/JHRI.1.1.Feil-Seifer

    Article  Google Scholar 

  23. Kato Y, Kanda T, Ishiguro H (2015) May i help you? - design of human-like polite approaching behavior. In: 2015 10th ACM/IEEE international conference on human-robot interaction (HRI) (pp. 35–42). IEEE

  24. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297. https://doi.org/10.1007/BF00994018

    Article  MATH  Google Scholar 

  25. Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238. https://doi.org/10.1109/TPAMI.2005.159

    Article  Google Scholar 

  26. Pattar SP, Coronado E, Ardila LR, Venture G (2019) Intention and engagement recognition for personalized human-robot interaction, an integrated and deep learning approach. In: 2019 IEEE 4th international conference on advanced robotics and mechatronics (ICARM) (pp. 93–98). IEEE. https://doi.org/10.1109/ICARM.2019.8834226

  27. Sidiropoulos GK, Papakostas GA, Lytridis C, Bazinas C, Kaburlasos VG et al (2020) Measuring engagement level in child-robot interaction using machine learning based data analysis. In: 2020 international conference on data analytics for business and industry: way towards a sustainable economy (ICDABI) (pp. 1–5). IEEE. https://doi.org/10.1109/ICDABI51230.2020.9325676

  28. Seber GA, Lee AJ (2012) Linear regression analysis. Wiley, New York

    MATH  Google Scholar 

  29. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366. https://doi.org/10.1016/0893-6080(89)90020-8

    Article  MATH  Google Scholar 

  30. Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324

    Article  MATH  Google Scholar 

  31. Zhang X, Yang X, Zhang W, Li G, Yu H (2021) Crowd emotion evaluation based on fuzzy inference of arousal and valence. Neurocomputing 445:194–205. https://doi.org/10.1016/j.neucom.2021.02.047

    Article  Google Scholar 

  32. Mehta D, Sotnychenko O, Mueller F, Xu W, Sridhar S et al (2018) Single-shot multi-person 3d pose estimation from monocular rgb. In: 2018 International Conference on 3D Vision (3DV) (pp. 120–130). IEEE. https://doi.org/10.1109/3DV.2018.00024

  33. Mollaret C, Mekonnen AA, Lerasle F, Ferrané I, Pinquier J et al (2016) A multi-modal perception based assistive robotic system for the elderly. Comput Vis Image Underst 149:78–97. https://doi.org/10.1016/j.cviu.2016.03.003

    Article  Google Scholar 

  34. Koo S, Kwon DS (2009) Recognizing human intentional actions from the relative movements between human and robot. In: RO-MAN 2009-The 18th IEEE international symposium on robot and human interactive communication (pp. 939–944). IEEE. https://doi.org/10.1109/ROMAN.2009.5326127

  35. Elman JL (1990) Finding structure in time. Cogn Sci 14(2):179–211. https://doi.org/10.1207/s15516709cog1402_1

    Article  Google Scholar 

  36. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  37. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

  38. Shi X, Chen Z, Wang H, Yeung DY, Wong WK, Woo WC (2015) Convolutional LSTM network: a machine learning approach for precipitation nowcasting. Adv Neural Inf Process Syst. Vol. 28

  39. Mamdani EH (1974) Applications of fuzzy algorithms for control of simple dynamic plant. Proc IEEE 121:1585–1588

    Google Scholar 

Download references

Acknowledgements

This work is supported by the Cooperative Project between universities in Chongqing and affiliated institutes of Chinese Academy of Sciences (No. HZ2021011), Youth Project of Science and Technology Research Program of Chongqing Education Commission of China (No. KJQN202101131), Graduate Innovation Project of Chongqing University of Technology (clgycx20201010) and Postdoctoral Science Foundation Program of Chongqing Science and Technology Bureau (No. CSTB2022NSCQ-BHX0674).

Author information

Authors and Affiliations

  1. College of Mechanical Engineering, Chongqing University of Technology, Chongqing, China

    Jian Bi, Fangchao Hu, Yujin Wang, Mingnan Luo & Miao He

Authors
  1. Jian Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fangchao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yujin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingnan Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Miao He
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation and data collection were performed by FH, YW and ML. Data analysis were performed by JB and MH. The first draft of the manuscript was written by JB and all authors commented on previous versions of the manuscript. Conceptualization and supervision was mainly performed by MH. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Miao He.

Ethics declarations

Conflict of interest

The author(s) declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

All of the authors confirm that there is no potential acts of misconduct in this work, and approve of the journal upholding the integrity of the scientific record.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent to publish

The authors affirm that human research participants provided informed consent for publication of the images in Figs. 1, 3, 10, 11 and 13.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bi, J., Hu, F., Wang, Y. et al. Human engagement intention intensity recognition method based on two states fusion fuzzy inference system. Intel Serv Robotics 16, 307–322 (2023). https://doi.org/10.1007/s11370-023-00464-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-023-00464-8

Keywords

Search

Navigation