Skip to main content
SpringerLink
Log in

High-Level Motor Planning Assessment During Performance of Complex Action Sequences in Humans and a Humanoid Robot

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Examining complex cognitive-motor performance in humanoid robots and humans can inform their interactions in a social context of team dynamics. Namely, the understanding of human cognitive-motor control and learning mechanisms can inform human motor behavior and also the development of intelligent controllers for robots when interacting with people. While prior humans and humanoid robot studies mainly examined motion planning, only a few have investigated high-level motor planning underlying action sequences for complex task execution. This sparse work has largely considered well-constrained problems using fairly simple performance assessment methods without detailed action sequence analyses. Here we qualitatively and quantitatively assess action sequences generated by humans and a humanoid robot during execution of two tasks providing various challenge levels and learning paradigms while offering flexible success criteria. The Levenshtein distance and its operators are adapted to the motor domain to provide a detailed performance assessment of action sequences by comparing them to a reference sequence (perfect sequence having a minimal number of actions). The results reveal that (i) humans produced a large variety of action sequences combining perfect and imperfect sequences while still reaching the task goal, whereas the robot generated perfect/near-perfect successful action sequences; (ii) the Levenshtein distance and the number of insertions provide reliable performance markers capable of differentiating perfect and imperfect sequences; (iii) the deletion operator is the most sensitive marker of action sequence failure. This work complements prior efforts for complex task performance assessment in humans and humanoid robots and has the potential to inform human–machine interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. While the direct comparison between humans and the robot is conducted here, this is secondary and rather of an exploratory nature in this work.

  2. For consistency between humans and the humanoid robot, here the term learning is employed in a general manner and reflects performance during the practice throughout the trials [27].

  3. For consistency with prior work the standard LD considering only these three operators was employed.

  4. The beginning, middle and end of the sequence were defined as 25% of the total number of actions forming the reference sequence (e.g., the beginning included the actions #1–2 and #1–4 for the HDDS and TOH task, respectively.

References

  1. Tenorth M, Ziegltrum J, Beetz M (2013) Automated alignment of specifications of everyday manipulation tasks. In: International conference on intelligent robots and systems (IROS), 2013 IEEE/RSJ, pp 5923–5928

  2. Tenorth M, Perzylo AC, Lafrenz R, Beetz M (2013) Representation and exchange of knowledge about actions, objects, and environments in the roboearth framework. IEEE Trans Autom Sci Eng 10(3):643–651

    Article  Google Scholar 

  3. Breazeal C, Buchsbaum D, Gray J, Gatenby D, Blumberg B (2005) Learning from and about others: towards using imitation to bootstrap the social understanding of others by robots. Artif Life 11(1–2):31–62

    Article  Google Scholar 

  4. Goel V, Grafman J (1995) Are the frontal lobes implicated in “planning” functions? Interpreting data from the Tower of Hanoi. Neuropsychologia 33(5):623–642

    Article  Google Scholar 

  5. Saint-Cyr JA, Taylor AE, Lang AE (1988) Procedural learning and neostriatal dysfunction in man. Brain 111(4):941–959

    Article  Google Scholar 

  6. Welsh MC, Huizinga M (2005) Tower of Hanoi disk-transfer task: influences of strategy knowledge and learning on performance. Learn Individ Differ 15(4):283–298

    Article  Google Scholar 

  7. Katz GE, Huang DW, Gentili RJ, Reggia JA (2016) Imitation learning as cause-effect reasoning. In: Steunebrink B, Wang P, Goertzel B (eds) Artificial general intelligence. AGI 2016, AGI 2016. Lecture notes in computer science, vol 9782. Springer, Cham

  8. Katz G, Huang DW, Hauge T, Gentili R, Reggia J (2017) A novel parsimonious cause-effect reasoning algorithm for robot imitation and plan recognition. IEEE Trans Cognit Dev Syst PP(99):1–17

    Google Scholar 

  9. Nathan K, Takayama L, Matarić M (2010) Communication and knowledge sharing in human–robot interaction and learning from demonstration. Neural Netw 23(8):1104–1112

    Google Scholar 

  10. Nikolaidis S, Ramakrishnan, Gu K, Shah J (2015) Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In: Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction. ACM

  11. Nikolaidis S, Hsu D, Srinivasa S (2017) Human-robot mutual adaptation in collaborative tasks: models and experiments. Int J Robot Res 36:618–634

    Article  Google Scholar 

  12. Paxton C, Jonathan F, Hundt A, Mutlu B, Hager GD (2017) User experience of the CoSTAR system for instruction of collaborative robots. arXiv preprint arXiv:1703.07890

  13. Steinfeld, Aaron, Fong T, Kaber D, Lewis M, Scholtz J, Schultz A, Goodrich M (2006) Common metrics for human–robot interaction. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction. ACM

  14. Wang N, Pynadath D, Hill S (2015) Building trust in a human robot team with automatically generated explanations. ITSEC paper, vol 15315 (2015), pp 1–12

  15. Zhang Y, Narayanan V, Chakraborti T, Kambhampati S (2015) A human factors analysis of proactive support in human–robot teaming. In: IEEE/RSJ international conference on. intelligent robots and systems (IROS)

  16. Levine SJ, Williams BC (2014) Concurrent plan recognition and execution for human-robot teams. In: ICAPS’14 proceedings of the twenty-fourth international conference on international conference on automated planning and scheduling, pp 490–498

  17. Shah J, Wiken J, Williams B, Breazeal C (2011) Improved human–robot team performance using chaski, a human-inspired plan execution system. In: Proceedings of the 6th international conference on Human–robot interaction. ACM

  18. Butchibabu A, Sparano-Huiban C, Sonenberg L, Shah J (2016) Implicit coordination strategies for effective team communication. Hum Factors 58(4):595–610

    Article  Google Scholar 

  19. Freedman RG, Shlomo Z (2017) Integration of planning with recognition for responsive interaction using classical planners. In: Thirty-first AAAI conference on artificial intelligence, San Francisco, CA, USA, pp 4581–4588

  20. Salter T, Dautenhahn K, Boekhorst R (2006) Learning about natural human–robot interaction styles. Robot Auton Syst 54(2):127–134

    Article  Google Scholar 

  21. Shah J, Breazeal C (2010) An empirical analysis of team coordination behaviors and action planning with application to human–robot teaming. Hum Factors 52(2):234–245

    Article  Google Scholar 

  22. Ho T, Oh SR, Kim H (2017) A parallel approximate string matching under Levenshtein distance on graphics processing units using warp-shuffle operations. PLoS ONE 12(10):e0186251

    Article  Google Scholar 

  23. Mohammad Y, Nishida T (2010) Controlling gaze with an embodied interactive control architecture. Appl Intell 32(2):148–163

    Article  Google Scholar 

  24. Mathieu JE, Hener TS, Goodwin GF, Salas E, Cannon-Bowers JA (2000) The influence of shared mental models on team process and performance. J Appl Psychol 85(2):273–283

    Article  Google Scholar 

  25. Scheutz M (2013) Computational mechanisms for mental models in human–robot interaction. In: Shumaker R (ed) Virtual augmented and mixed reality. Designing and developing augmented and virtual environments. VAMR 2013. Lecture notes in computer science, vol 8021. Springer, Berlin

    Google Scholar 

  26. Sheridan TB (2016) Human–robot interaction: status and challenges. Hum Factors 58(4):525–532

    Article  Google Scholar 

  27. Magill RA, Anderson DI (2017) Motor learning and control: Concepts and applications, 11th edn. McGraw-Hill, Boston

    Google Scholar 

  28. Huang DW, Katz GE, Langsfeld JD, Gentili RJ, Reggia JA (2015) A virtual demonstrator environment for robot imitation learning. In: IEEE international conference on technologies for practical robot applications (TePRA), Woburn, MA, USA, pp 1–6

  29. Huang DW, Katz GE, Langsfeld JD, Oh H, Gentili RJ, Reggia JA (2015) An object-centric paradigm for robot programming by demonstration. In: International conference on augmented cognition, Springer, Los Angeles, CA, USA, pp 745–756

  30. Palmer DW (1996) Exploring recursion with variations on the Towers of Hanoi. J Comput Sci Coll 12(96):59–66

    Google Scholar 

  31. Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions, and reversals. Sov Phys Dokl 10(8):707–710

    MathSciNet  Google Scholar 

  32. Yujian L, Bo L (2007) A normalized Levenshtein distance metric. IEEE Trans Pattern Anal Mach Intell 29(6):1091–1095

    Article  Google Scholar 

  33. Wagner RA, Fischer MJ (1974) The string-to-string correction problem. J ACM (JACM) 21(1):168–173

    Article  MathSciNet  MATH  Google Scholar 

  34. Deese J, Kaufman RA (1957) Serial effects in recall of unorganized and sequentially organized verbal material. J Exp Psychol 54:180–187

    Article  Google Scholar 

  35. Murdoch BB (1962) The serial position effect of free recall. J Exp Psychol 64(5):482–488

    Article  Google Scholar 

  36. Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodol) 57(1):289–300

    MathSciNet  MATH  Google Scholar 

  37. Morrison AB, Conway AR, Chein JM (2014) Primacy and recency effects as indices of the focus of attention. Front Hum Neurosci 24(8):6 (eCollection 2014)

    Google Scholar 

  38. Gentili RJ, Jaquess KJ, Shuggi IM, Oh H, Lo LC, Tan YY, Domingues CA, Blanco JA, Rietschel JC, Miller MW, Hatfield BD (2018) Combined assessment of attentional reserve and cognitive effort under various levels of challenge with a dry EEG system. Psychophysiology 55(6):e13059

    Article  Google Scholar 

  39. Shaw EP, Rietschel JC, Hendershot BD, Pruziner AL, Miller MW, Hatfield BD, Gentili RJ (2018) Measurement of attentional reserve and mental effort for cognitive workload assessment under various task demands during dual-task walking. Biol Psychol 134:39–51

    Article  Google Scholar 

  40. Shuggi IM, Oh H, Shewokis PA, Gentili RJ (2017) Mental workload and motor performance dynamics during practice of reaching movements under various levels of task difficulty. Neuroscience 360:166–179

    Article  Google Scholar 

  41. Shuggi IM, Shewokis PA, Herrmann JW, Gentili RJ (2017) Changes in motor performance and mental workload during learning of reaching movements: a team dynamics perspective. Exp Brain Res 236(2):433–451

    Article  Google Scholar 

  42. Cowan N (2001) The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci 24(1):87–185

    Article  Google Scholar 

  43. Omote K, Feussner H, Ungeheuer A, Arbter K, Wei GQ, Siewert JR, Hirzinger G (1999) Self-guided robotic camera control for laparoscopic surgery compared with human camera control. Am J Surg 177(4):321–324

    Article  Google Scholar 

  44. Gorbenko A, Popov V (2015) On the adjustment of the weights of the Levenshtein distance for the description of sequences of rhythmic motor primitives. Contemp Eng Sci 8(18):835–840

    Article  Google Scholar 

  45. Holm L, Karampela O, Ullén F, Madison G (2017) Executive control and working memory are involved in sub-second repetitive motor timing. Exp Brain Res 235:787–798

    Article  Google Scholar 

  46. Losey DP, McDonald CG, Battaglia E, O’Malley MK (2018) A review of intent detection, arbitration, and communication aspects of shared control for physical human–robot interaction. Appl Mech Rev 70(1):1–19

    Google Scholar 

  47. Miller MW, Presacco A, Groman LJ, Bur S, Rietschel JC, Gentili RJ, McDonald CG, Iso-Ahola SE, Hatfield BD (2014) The effects of team environment on cerebral cortical processes and attentional reserve. Sport Exerc Perform Psychol 3(1):61–74

    Article  Google Scholar 

  48. Ahrndt S, Albayrak S (2016) Joint human-agent activities: challenges and definition, vol 9872. Springer, Cham, pp 105–112

    Google Scholar 

  49. Hanna N, Richards D (2018) The impact of multimodal communication on a shared mental model, trust, and commitment in human–intelligent virtual agent teams. Multimodal Technologies Interact 2:48

    Article  Google Scholar 

  50. Honig S, Oron-Gilad T (2018) Understanding and resolving failures in human-robot interaction: literature review and model development. Front Psychol 9:861

    Article  Google Scholar 

  51. Gillespie RB (2001) A general framework for cobot control. IEEE Trans Robot Autom 17(4):391–400

    Article  Google Scholar 

  52. Boehm- P, Ghasemi AH, O’Modhrain S, Jayakumar O, Gillespie RB (2016) Architectures for shared control of vehicle steering. IFAC-PapersOnLine 49–19:639–644

    Article  Google Scholar 

  53. Fitter NT, Kuchenbecker KJ (2016) Qualitative user reactions to a hand-clapping humanoid robot. In: Agah A, Cabibihan JJ, Howard A, Salichs M, He H (eds) Social robotics. ICSR 2016. Lecture notes in computer science, vol 9979. Springer, Cham

  54. Vakil E, Lev-Ran Galon C (2014) Baseline performance and learning rate of conceptual and perceptual skill-learning tasks: the effect of moderate to severe traumatic brain injury. J Clin Exp Neuropsychol 36(5):447–454

    Article  Google Scholar 

  55. Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7(9):907–915

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by The Office of Naval Research (N00014-19-1-2044).

Author information

Author notes

  1. Theresa C. Hauge and Garrett E. Katz have contributed equally to this work and are co-first authors.

Authors and Affiliations

  1. Cognitive Motor Neuroscience Laboratory, Department of Kinesiology, School of Public Health, University of Maryland, 4200 Valley Drive, College Park, MD, 20742, USA

    Theresa C. Hauge & Rodolphe J. Gentili

  2. Department of Electrical Engineering and Computer Science, Syracuse University, New York, 13244, USA

    Garrett E. Katz

  3. Department of Computer Science,, University of Maryland, College Park, MD, 20742, USA

    Gregory P. Davis, Di-Wei Huang & James A. Reggia

  4. Institute for Advanced Computer Studies, University of Maryland, College Park, MD, 20742, USA

    James A. Reggia

  5. Neuroscience and Cognitive Science Program, University of Maryland, College Park, MD, USA

    James A. Reggia & Rodolphe J. Gentili

  6. Maryland Robotics Center, University of Maryland, College Park, MD, USA

    James A. Reggia & Rodolphe J. Gentili

Authors
  1. Theresa C. Hauge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Garrett E. Katz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gregory P. Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Di-Wei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. James A. Reggia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Rodolphe J. Gentili
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rodolphe J. Gentili.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 250 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hauge, T.C., Katz, G.E., Davis, G.P. et al. High-Level Motor Planning Assessment During Performance of Complex Action Sequences in Humans and a Humanoid Robot. Int J of Soc Robotics 13, 981–998 (2021). https://doi.org/10.1007/s12369-020-00685-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00685-2

Keywords

Search

Navigation