Skip to main content
SpringerLink
Log in

Assessing task understanding in remote ultrasound diagnosis via gesture analysis

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

This work presents a gesture-based approach to estimate task understanding and performance during remote ultrasound tasks. Our approach is comprised of two main components. The first component uses the Multi-Agent Gestural Instruction Comparer (MAGIC) framework to represent and compare the gestures performed by collaborators. Through MAGIC, gestures can be compared based in their morphology, semantics, and pragmatics. The second component computes the Physical Instructions Assimilation (PIA) metric, a score representing how well are gestures being used to communicate and execute physical instructions. To evaluate our hypothesis, 20 participants performed a remote ultrasound task consisting of three subtasks: vessel detection, blood extraction, and foreign body detection. MAGIC’s gesture comparison approaches were compared against two other approaches based on how well they replicated human-annotated gestures matchings. Our approach outperformed the others, agreeing with the human baseline over 76% of the times. Subsequently, a correlation analysis was performed to compare PIA’s task understanding insights with those of three other metrics: error rate, idle time rate, and task completion percentage. Significant correlations (\(p\,\le \,0.04\)) were found between PIA and all the other metrics, positioning PIA as an effective metric for task understanding estimation. Finally, post-experiment questionnaires were used to subjectively evaluate the participants’ perceived understanding. The PIA score was found to be significantly correlated with the participants’ overall task understanding (\(p\le 0.05\)), hinting to the relation between the assimilation of physical instructions and self-perceived understanding. These results demonstrate that gestures an be used to estimate task understanding in remote ultrasound tasks, which can improve how these tasks are performed and assessed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abuhamad A, Minton KK, Benson CB, Chudleigh T, Crites L, Doubilet PM, Driggers R, Lee W, Mann KV, Perez JJ et al (2018) Obstetric and gynecologic ultrasound curriculum and competency assessment in residency training programs: consensus report. Am J Obstet Gynecol 218(1):29–67

    Article  Google Scholar 

  2. Aldekhyl S, Cavalcanti RB, Naismith LM (2018) Cognitive load predicts point-of-care ultrasound simulator performance. Perspect Med Educ 7(1):23–32

    Article  Google Scholar 

  3. Alibali MW, Kita S, Young AJ (2000) Gesture and the process of speech production: we think, therefore we gesture. Lang Cognit Process 15(6):593–613

    Article  Google Scholar 

  4. Amini R, Kartchner JZ, Stolz LA, Biffar D, Hamilton AJ, Adhikari S (2015) A novel and inexpensive ballistic gel phantom for ultrasound training. World J Emerg Med 6(3):225

    Article  Google Scholar 

  5. Aussems S, Kita S (2019) Seeing iconic gestures while encoding events facilitates childrens memory of these events. Child Dev 90(4):1123–1137

    Article  Google Scholar 

  6. Barmby P, Harries T, Higgins S, Suggate J (2007) How can we assess mathematical understanding. In: Proceedings of the 31st conference of the international group for the psychology of mathematics education, vol 2. ERIC, pp 41–48

  7. Broaders SC, Cook SW, Mitchell Z, Goldin-Meadow S (2007) Making children gesture brings out implicit knowledge and leads to learning. J Exp Psychol Gen 136(4):539

    Article  Google Scholar 

  8. Cheng J, Zhou W, Lei X, Adamo N, Benes B (2020) The effects of body gestures and gender on viewer’s perception of animated pedagogical agent’s emotions. In: International conference on human–computer interaction. Springer, pp 169–186

  9. Chung YA, Weng WH, Tong S, Glass J (2019) Towards unsupervised speech-to-text translation. In: ICASSP 2019–2019 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 7170–7174

  10. Clark HH, Brennan SE et al (1991) Grounding in communication. Perspect Soc Shared Cognit 13(1991):127–149

    Article  Google Scholar 

  11. Clark HH, Schaefer EF (1987) Collaborating on contributions to conversations. Lang Cognit Process 2(1):19–41

    Article  Google Scholar 

  12. Cook SW, Yip TK, Goldin-Meadow S (2012) Gestures, but not meaningless movements, lighten working memory load when explaining math. Lang Cognit Process 27(4):594–610

    Article  Google Scholar 

  13. Cornia M, Baraldi L, Serra G, Cucchiara R (2018) Paying more attention to saliency: image captioning with saliency and context attention. ACM Trans Multimed Comput Commun Appl (TOMM) 14(2):1–21

    Article  Google Scholar 

  14. Czyzyk J, Mesnier MP, Moré JJ (1998) The neos server. IEEE J Comput Sci Eng 5(3):68–75

    Article  Google Scholar 

  15. Dekker S (2017) The field guide to understanding‘human error’. CRC Press, London

  16. Dolan ED (2001) The neos server 4.0 administrative guide. Technical Memorandum ANL/MCS-TM-250, Mathematics and Computer Science Division, Argonne National. Laboratory

  17. Dufurrena Q, Ullah KI, Taub E, Leszczuk C, Ahmad S (2020) Feasibility and clinical implications of remotely guided ultrasound examinations. Aerosp Med Human Perform 91(7):592–596

    Article  Google Scholar 

  18. Évain A, Argelaguet F, Strock A, Roussel N, Casiez G, Lécuyer A (2016) Influence of error rate on frustration of bci users. In: Proceedings of the international working conference on advanced visual interfaces, pp 248–251

  19. Fillauer JP, Bolden J, Jacobson M, Partlow BH, Benavides A, Shultz JN (2020) Examining the effects of frustration on working memory capacity. Appl Cogn Psychol 34(1):50–63

    Article  Google Scholar 

  20. Fruchter R, Cavallin H (2011) Attention and engagement of remote team members in collaborative multimedia environments. In: Computing in civil engineering (2011). American Society of Civil Engineers, pp 875–882

  21. Fussell SR, Setlock LD, Yang J, Ou J, Mauer E, Kramer AD (2004) Gestures over video streams to support remote collaboration on physical tasks. Hum Comput Interact 19(3):273–309

    Article  Google Scholar 

  22. Gergle D, Kraut RE, Fussell SR (2004) Action as language in a shared visual space. In: Proceedings of the 2004 ACM conference on Computer supported cooperative work, pp 487–496

  23. Goldin-Meadow S, Beilock SL (2010) Actionś influence on thought: the case of gesture. Perspect Psychol Sci 5(6):664–674

    Article  Google Scholar 

  24. Goutte C, Gaussier E(2005) A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. In: European conference on information retrieval. Springer, pp 345–359

  25. Gropp W, Moré JJ (1997) Optimization environments and the neos server. In: Buhman MD, Iserle A (eds) Approximation theory and optimization. Cambridge University Press, Cambridge, pp 167–182

    MATH  Google Scholar 

  26. Haraldsen HM, Solstad BE, Ivarsson A, Halvari H, Abrahamsen FE (2019) Change in basic need frustration in relation to perfectionism, anxiety and performance in elite junior performers. Scand J Med Sci Sports 30(4):754–765

    Article  Google Scholar 

  27. Hart SG (2006) Nasa-task load index (nasa-tlx); 20 years later. In: Proceedings of the human factors and ergonomics society annual meeting, vol 50. Sage publications, Los Angeles, pp 904–908

  28. Hoffman G (2019) Evaluating fluency in human–robot collaboration. IEEE Trans Hum Mach Syst 49(3):209–218

    Article  Google Scholar 

  29. Huang W, Kim S, Billinghurst M, Alem L (2019) Sharing hand gesture and sketch cues in remote collaboration. J Vis Commun Image Represent 58:428–438

    Article  Google Scholar 

  30. Kim S, Lee G, Huang W, Kim H, Woo W, Billinghurst M (2019) Evaluating the combination of visual communication cues for hmd-based mixed reality remote collaboration. In: Proceedings of the 2019 CHI conference on human factors in computing systems. ACM, p 173

  31. Kirk D, Crabtree A, Rodden T (2005) Ways of the hands. In: ECSCW 2005. Springer, pp 1–21

  32. Kirk D, Rodden T, Fraser DS (2007) Turn it this way: grounding collaborative action with remote gestures. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 1039–1048

  33. Kirkpatrick AW, McKee JL, McBeth PB, Ball CG, LaPorta A, Broderick T, Leslie T, King D, Beatty HEW, Keillor J et al (2017) The damage control surgery in austere environments research group (dcsaerg): a dynamic program to facilitate real-time telementoring/telediagnosis to address exsanguination in extreme and austere environments. J Trauma Acute Care Surg 83(1):S156–S163

    Article  Google Scholar 

  34. Knoblich G, Sebanz N (2006) The social nature of perception and action. Curr Dir Psychol Sci 15(3):99–104

    Article  Google Scholar 

  35. Laurent DABS, Niazi AU, Cunningham MS, Jaeger M, Abbas S, McVicar J, Chan VW (2014) A valid and reliable assessment tool for remote simulation-based ultrasound-guided regional anesthesia. Reg Anesth Pain Med 39(6):496–501

    Article  Google Scholar 

  36. Liu R, Holden MS (2020) Kinematics data representations for skills assessment in ultrasound-guided needle insertion. In: Hu Y, Licandro R, Noble JA, Hutter J, Aylward S, Melbourne A, Turk EA, Barrena JT (eds) Medical ultrasound, and preterm, perinatal and paediatric image analysis. Springer, pp 189–198

  37. Madapana N, Wachs J (2017) Zsgl: zero shot gestural learning. In: Proceedings of the 19th ACM international conference on multimodal interaction. ACM, pp 331–335

  38. Martinez-Moyano I (2006) Exploring the dynamics of collaboration in interorganizational settings. In: Creating a culture of collaboration: the international association of facilitators handbook, vol 4, p 69

  39. McBeth PB, Crawford I, Blaivas M, Hamilton T, Musselwhite K, Panebianco N, Melniker L, Ball CG, Gargani L, Gherdovich C et al (2011) Simple, almost anywhere, with almost anyone: remote low-cost telementored resuscitative lung ultrasound. J Trauma Acute Care Surg 71(6):1528–1535

    Article  Google Scholar 

  40. McBeth PB, Hamilton T, Kirkpatrick AW (2010) Cost-effective remote iphone-teathered telementored trauma telesonography. J Trauma Acute Care Surg 69(6):1597–1599

    Article  Google Scholar 

  41. Mitragotri S (2005) Healing sound: the use of ultrasound in drug delivery and other therapeutic applications. Nat Rev Drug Discovery 4(3):255–260

    Article  Google Scholar 

  42. Molnar-Szakacs I, Wu AD, Robles FJ, Iacoboni M (2007) Do you see what i mean? Corticospinal excitability during observation of culture-specific gestures. PLoS One 2(7):e626

    Article  Google Scholar 

  43. Nelson BP, Melnick ER, Li J (2011) Portable ultrasound for remote environments, part i: feasibility of field deployment. J Emerg Med 40(2):190–197

    Article  Google Scholar 

  44. Olszynski P, Heslop C, Atkinson P, Lewis D, Kim DJ, Pham C, Ritcey B (2020) Ultrasound at the point of care-grown up and moving out! Canad J Emerg Med 22(1):1–2

    Article  Google Scholar 

  45. Pearson K (1895) Vii. note on regression and inheritance in the case of two parents. Proc R Soc Lond 58(347–352):240–242

    Google Scholar 

  46. Ping RM, Goldin-Meadow S, Beilock SL (2014) Understanding gesture: is the listener’s motor system involved? J Exp Psychol General 143(1):195

  47. Rojas-Muñoz E, Wachs JP (2019) MAGIC: a fundamental framework for gesture representation, comparison and assessment. In: 2019 14th IEEE international conference on automatic face and gesture recognition (FG 2019). IEEE, pp 1–8

  48. Rojas-Muñoz E, Wachs JP (2020) Beyond MAGIC: matching collaborative gestures using an optimization-based approach. In: 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020)(FG), pp 296–303

  49. Rojas-Muñoz E, Wachs JP (2020) The MAGIC of e-health: a gesture-based approach to estimate understanding and performance in remote ultrasound tasks. In: 2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020)(FG), pp 314–318

  50. Roth WM (2001) Gestures: their role in teaching and learning. Rev Educ Res 71(3):365–392

    Article  Google Scholar 

  51. Sacks H, Schegloff EA, Jefferson G (1978) A simplest systematics for the organization of turn taking for conversation. In: Studies in the organization of conversational interaction. Elsevier, pp 7–55

  52. Schlager D, Sanders AB, Wiggins D, Boren W (1991) Ultrasound for the detection of foreign bodies. Ann Emerg Med 20(2):189–191

    Article  Google Scholar 

  53. Schubotz L, Özyürek A, Holler J (2019) Age-related differences in multimodal recipient design: younger, but not older adults, adapt speech and co-speech gestures to common ground. Lang Cognit Neurosci 34(2):254–271

    Article  Google Scholar 

  54. Sebanz N, Knoblich G, Prinz W (2003) Representing others’ actions: just like one’s own? Cognition 88(3):B11–B21

  55. Tang Y, Cheng S, Yang Y, Xiang X, Wang L, Zhang L, Qiu L (2020) Ultrasound assessment in psoriatic arthritis (psa) and psoriasis vulgaris (non-psa): which sites are most commonly involved and what features are more important in psa? Quant Imaging Med Surg 10(1):86

    Article  Google Scholar 

  56. Thorn S, Gopalasingam N, Bendtsen TF, Knudsen L, Sloth E (2016) A technique for ultrasound-guided blood sampling from a dry and gel-free puncture area. J Vasc Access 17(3):265–268

    Article  Google Scholar 

  57. Vatsvåg V, Todnem K, Næsheim T, Cathcart J, Kerr D, Oveland NP (2020) Offshore telementored ultrasound: a quality assessment study. Ultrasound J 12(1):1–12

    Article  Google Scholar 

  58. White R, Gunstone R (2014) Probing understanding. Routledge, London

    Book  Google Scholar 

  59. Wong I, Jayatilleke T, Kendall R, Atkinson P (2011) Feasibility of a focused ultrasound training programme for medical undergraduate students. Clin Teach 8(1):3–7

    Article  Google Scholar 

  60. Wydo S, Seamon M, Melanson S, Thomas P, Bahner DP, Stawicki SP (2016) Portable ultrasound in disaster triage: a focused review. Eur J Trauma Emerg Surg 42(2):151–159

    Article  Google Scholar 

  61. Yeo LM, Tzeng YT (2020) Cognitive effect of tracing gesture in the learning from mathematics worked examples. Int J Sci Math Educ 18(4):733–751

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Visualization, Texas A&M University, 3137 TAMU, College Station, TX, 77840, USA

    Edgar Rojas-Muñoz

  2. School of Industrial Engineering, Purdue University, 315 N. Grant Street, West Lafayette, IN, 47907, USA

    Juan P. Wachs

Authors
  1. Edgar Rojas-Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan P. Wachs
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Juan P. Wachs.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rojas-Muñoz, E., Wachs, J.P. Assessing task understanding in remote ultrasound diagnosis via gesture analysis. Pattern Anal Applic 24, 1489–1500 (2021). https://doi.org/10.1007/s10044-021-01027-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-021-01027-2

Keywords

Search

Navigation