Examining complex cognitive-motor performance in humanoid robots and humans can inform their interactions in a social context of team dynamics. Namely, the understanding of human cognitive-motor control and learning mechanisms can inform human motor behavior and also the development of intelligent controllers for robots when interacting with people. While prior humans and humanoid robot studies mainly examined motion planning, only a few have investigated high-level motor planning underlying action sequences for complex task execution. This sparse work has largely considered well-constrained problems using fairly simple performance assessment methods without detailed action sequence analyses. Here we qualitatively and quantitatively assess action sequences generated by humans and a humanoid robot during execution of two tasks providing various challenge levels and learning paradigms while offering flexible success criteria. The Levenshtein distance and its operators are adapted to the motor domain to provide a detailed performance assessment of action sequences by comparing them to a reference sequence (perfect sequence having a minimal number of actions). The results reveal that (i) humans produced a large variety of action sequences combining perfect and imperfect sequences while still reaching the task goal, whereas the robot generated perfect/near-perfect successful action sequences; (ii) the Levenshtein distance and the number of insertions provide reliable performance markers capable of differentiating perfect and imperfect sequences; (iii) the deletion operator is the most sensitive marker of action sequence failure. This work complements prior efforts for complex task performance assessment in humans and humanoid robots and has the potential to inform human–machine interactions.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
While the direct comparison between humans and the robot is conducted here, this is secondary and rather of an exploratory nature in this work.
For consistency between humans and the humanoid robot, here the term learning is employed in a general manner and reflects performance during the practice throughout the trials .
For consistency with prior work the standard LD considering only these three operators was employed.
The beginning, middle and end of the sequence were defined as 25% of the total number of actions forming the reference sequence (e.g., the beginning included the actions #1–2 and #1–4 for the HDDS and TOH task, respectively.
Tenorth M, Ziegltrum J, Beetz M (2013) Automated alignment of specifications of everyday manipulation tasks. In: International conference on intelligent robots and systems (IROS), 2013 IEEE/RSJ, pp 5923–5928
Tenorth M, Perzylo AC, Lafrenz R, Beetz M (2013) Representation and exchange of knowledge about actions, objects, and environments in the roboearth framework. IEEE Trans Autom Sci Eng 10(3):643–651
Breazeal C, Buchsbaum D, Gray J, Gatenby D, Blumberg B (2005) Learning from and about others: towards using imitation to bootstrap the social understanding of others by robots. Artif Life 11(1–2):31–62
Goel V, Grafman J (1995) Are the frontal lobes implicated in “planning” functions? Interpreting data from the Tower of Hanoi. Neuropsychologia 33(5):623–642
Saint-Cyr JA, Taylor AE, Lang AE (1988) Procedural learning and neostriatal dysfunction in man. Brain 111(4):941–959
Welsh MC, Huizinga M (2005) Tower of Hanoi disk-transfer task: influences of strategy knowledge and learning on performance. Learn Individ Differ 15(4):283–298
Katz GE, Huang DW, Gentili RJ, Reggia JA (2016) Imitation learning as cause-effect reasoning. In: Steunebrink B, Wang P, Goertzel B (eds) Artificial general intelligence. AGI 2016, AGI 2016. Lecture notes in computer science, vol 9782. Springer, Cham
Katz G, Huang DW, Hauge T, Gentili R, Reggia J (2017) A novel parsimonious cause-effect reasoning algorithm for robot imitation and plan recognition. IEEE Trans Cognit Dev Syst PP(99):1–17
Nathan K, Takayama L, Matarić M (2010) Communication and knowledge sharing in human–robot interaction and learning from demonstration. Neural Netw 23(8):1104–1112
Nikolaidis S, Ramakrishnan, Gu K, Shah J (2015) Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In: Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction. ACM
Nikolaidis S, Hsu D, Srinivasa S (2017) Human-robot mutual adaptation in collaborative tasks: models and experiments. Int J Robot Res 36:618–634
Paxton C, Jonathan F, Hundt A, Mutlu B, Hager GD (2017) User experience of the CoSTAR system for instruction of collaborative robots. arXiv preprint arXiv:1703.07890
Steinfeld, Aaron, Fong T, Kaber D, Lewis M, Scholtz J, Schultz A, Goodrich M (2006) Common metrics for human–robot interaction. In: Proceedings of the 1st ACM SIGCHI/SIGART conference on human-robot interaction. ACM
Wang N, Pynadath D, Hill S (2015) Building trust in a human robot team with automatically generated explanations. ITSEC paper, vol 15315 (2015), pp 1–12
Zhang Y, Narayanan V, Chakraborti T, Kambhampati S (2015) A human factors analysis of proactive support in human–robot teaming. In: IEEE/RSJ international conference on. intelligent robots and systems (IROS)
Levine SJ, Williams BC (2014) Concurrent plan recognition and execution for human-robot teams. In: ICAPS’14 proceedings of the twenty-fourth international conference on international conference on automated planning and scheduling, pp 490–498
Shah J, Wiken J, Williams B, Breazeal C (2011) Improved human–robot team performance using chaski, a human-inspired plan execution system. In: Proceedings of the 6th international conference on Human–robot interaction. ACM
Butchibabu A, Sparano-Huiban C, Sonenberg L, Shah J (2016) Implicit coordination strategies for effective team communication. Hum Factors 58(4):595–610
Freedman RG, Shlomo Z (2017) Integration of planning with recognition for responsive interaction using classical planners. In: Thirty-first AAAI conference on artificial intelligence, San Francisco, CA, USA, pp 4581–4588
Salter T, Dautenhahn K, Boekhorst R (2006) Learning about natural human–robot interaction styles. Robot Auton Syst 54(2):127–134
Shah J, Breazeal C (2010) An empirical analysis of team coordination behaviors and action planning with application to human–robot teaming. Hum Factors 52(2):234–245
Ho T, Oh SR, Kim H (2017) A parallel approximate string matching under Levenshtein distance on graphics processing units using warp-shuffle operations. PLoS ONE 12(10):e0186251
Mohammad Y, Nishida T (2010) Controlling gaze with an embodied interactive control architecture. Appl Intell 32(2):148–163
Mathieu JE, Hener TS, Goodwin GF, Salas E, Cannon-Bowers JA (2000) The influence of shared mental models on team process and performance. J Appl Psychol 85(2):273–283
Scheutz M (2013) Computational mechanisms for mental models in human–robot interaction. In: Shumaker R (ed) Virtual augmented and mixed reality. Designing and developing augmented and virtual environments. VAMR 2013. Lecture notes in computer science, vol 8021. Springer, Berlin
Sheridan TB (2016) Human–robot interaction: status and challenges. Hum Factors 58(4):525–532
Magill RA, Anderson DI (2017) Motor learning and control: Concepts and applications, 11th edn. McGraw-Hill, Boston
Huang DW, Katz GE, Langsfeld JD, Gentili RJ, Reggia JA (2015) A virtual demonstrator environment for robot imitation learning. In: IEEE international conference on technologies for practical robot applications (TePRA), Woburn, MA, USA, pp 1–6
Huang DW, Katz GE, Langsfeld JD, Oh H, Gentili RJ, Reggia JA (2015) An object-centric paradigm for robot programming by demonstration. In: International conference on augmented cognition, Springer, Los Angeles, CA, USA, pp 745–756
Palmer DW (1996) Exploring recursion with variations on the Towers of Hanoi. J Comput Sci Coll 12(96):59–66
Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions, and reversals. Sov Phys Dokl 10(8):707–710
Yujian L, Bo L (2007) A normalized Levenshtein distance metric. IEEE Trans Pattern Anal Mach Intell 29(6):1091–1095
Wagner RA, Fischer MJ (1974) The string-to-string correction problem. J ACM (JACM) 21(1):168–173
Deese J, Kaufman RA (1957) Serial effects in recall of unorganized and sequentially organized verbal material. J Exp Psychol 54:180–187
Murdoch BB (1962) The serial position effect of free recall. J Exp Psychol 64(5):482–488
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodol) 57(1):289–300
Morrison AB, Conway AR, Chein JM (2014) Primacy and recency effects as indices of the focus of attention. Front Hum Neurosci 24(8):6 (eCollection 2014)
Gentili RJ, Jaquess KJ, Shuggi IM, Oh H, Lo LC, Tan YY, Domingues CA, Blanco JA, Rietschel JC, Miller MW, Hatfield BD (2018) Combined assessment of attentional reserve and cognitive effort under various levels of challenge with a dry EEG system. Psychophysiology 55(6):e13059
Shaw EP, Rietschel JC, Hendershot BD, Pruziner AL, Miller MW, Hatfield BD, Gentili RJ (2018) Measurement of attentional reserve and mental effort for cognitive workload assessment under various task demands during dual-task walking. Biol Psychol 134:39–51
Shuggi IM, Oh H, Shewokis PA, Gentili RJ (2017) Mental workload and motor performance dynamics during practice of reaching movements under various levels of task difficulty. Neuroscience 360:166–179
Shuggi IM, Shewokis PA, Herrmann JW, Gentili RJ (2017) Changes in motor performance and mental workload during learning of reaching movements: a team dynamics perspective. Exp Brain Res 236(2):433–451
Cowan N (2001) The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci 24(1):87–185
Omote K, Feussner H, Ungeheuer A, Arbter K, Wei GQ, Siewert JR, Hirzinger G (1999) Self-guided robotic camera control for laparoscopic surgery compared with human camera control. Am J Surg 177(4):321–324
Gorbenko A, Popov V (2015) On the adjustment of the weights of the Levenshtein distance for the description of sequences of rhythmic motor primitives. Contemp Eng Sci 8(18):835–840
Holm L, Karampela O, Ullén F, Madison G (2017) Executive control and working memory are involved in sub-second repetitive motor timing. Exp Brain Res 235:787–798
Losey DP, McDonald CG, Battaglia E, O’Malley MK (2018) A review of intent detection, arbitration, and communication aspects of shared control for physical human–robot interaction. Appl Mech Rev 70(1):1–19
Miller MW, Presacco A, Groman LJ, Bur S, Rietschel JC, Gentili RJ, McDonald CG, Iso-Ahola SE, Hatfield BD (2014) The effects of team environment on cerebral cortical processes and attentional reserve. Sport Exerc Perform Psychol 3(1):61–74
Ahrndt S, Albayrak S (2016) Joint human-agent activities: challenges and definition, vol 9872. Springer, Cham, pp 105–112
Hanna N, Richards D (2018) The impact of multimodal communication on a shared mental model, trust, and commitment in human–intelligent virtual agent teams. Multimodal Technologies Interact 2:48
Honig S, Oron-Gilad T (2018) Understanding and resolving failures in human-robot interaction: literature review and model development. Front Psychol 9:861
Gillespie RB (2001) A general framework for cobot control. IEEE Trans Robot Autom 17(4):391–400
Boehm- P, Ghasemi AH, O’Modhrain S, Jayakumar O, Gillespie RB (2016) Architectures for shared control of vehicle steering. IFAC-PapersOnLine 49–19:639–644
Fitter NT, Kuchenbecker KJ (2016) Qualitative user reactions to a hand-clapping humanoid robot. In: Agah A, Cabibihan JJ, Howard A, Salichs M, He H (eds) Social robotics. ICSR 2016. Lecture notes in computer science, vol 9979. Springer, Cham
Vakil E, Lev-Ran Galon C (2014) Baseline performance and learning rate of conceptual and perceptual skill-learning tasks: the effect of moderate to severe traumatic brain injury. J Clin Exp Neuropsychol 36(5):447–454
Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7(9):907–915
This work was supported by The Office of Naval Research (N00014-19-1-2044).
Conflict of interest
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Hauge, T.C., Katz, G.E., Davis, G.P. et al. High-Level Motor Planning Assessment During Performance of Complex Action Sequences in Humans and a Humanoid Robot. Int J of Soc Robotics 13, 981–998 (2021). https://doi.org/10.1007/s12369-020-00685-2
- Cognitive-motor control and learning
- High-level motor planning
- Humanoid robot
- Action sequence