Abstract
Background
Hand motion analysis by video recording during surgery has potential for evaluation of surgical performance. The aim was to identify how technical skill during open surgery can be measured unobtrusively by video recording during a surgical procedure. We hypothesized that procedural-step timing, hand movements, instrument use and Shannon entropy differ with expertise and training and are concordant with a performance-based validated individual procedure score.
Methods
Surgeon and non-surgeon participants with varying training and levels of expertise were video recorded performing axillary artery exposure and control (AA) on un-preserved cadavers. Color-coded gloves permitted motion-tracking and automated extraction of entropy data from recordings. Timing and instrument-use metrics were obtained through observational video reviews. Shannon entropy measured speed, acceleration and direction by computer-vision algorithms. Findings were compared with individual procedure score for AA performance
Results
Experts had lowest entropy values, idle time, active time and shorter time to divide pectoralis minor, using fewer instruments. Residents improved with training, without reaching expert levels, and showed deterioration 12–18 months later. Individual procedure scores mirrored these results. Non-surgeons differed substantially.
Conclusions
Hand motion entropy and timing metrics discriminate levels of surgical skill and training, and these findings are congruent with individual procedure score evaluations. These measures can be collected using consumer-level cameras and analyzed automatically with free software. Hand motion with video timing data may have widespread application to evaluate resident performance and can contribute to the range of evaluation and testing modalities available to educators, training course designers and surgical quality assurance programs.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Written or oral examination performances can be unreliable indicators of the real-world technical performance of surgeons. Current assessment of technical skills is based on subjective opinions of senior colleagues [1, 2]. Surgical residents’ technical skill is typically evaluated from observations by experienced mentors during training; however, this process is time-consuming, labor-intensive and may include evaluator biases. Three technical performance evaluations are currently validated, the most often used is the objective structured assessment of technical skills (OSATS) [3]. Procedure-based assessments (PBA) and direct observation of practical skills (DOPS) are integrated within the Intercollegiate Surgical Curriculum Programme online platform and aim to assess trainees’ performance in practical surgical skills [4]. The individual procedure score (IPS) has been validated for vascular hemorrhage control and non-vascular trauma procedures [5,6,7,8]. OSATS, PBA and IPS were found to identify surgeons, at all levels of seniority, who are in need of remediation of technical skills for open surgery [3,4,5]; however, all these evaluations require significant resources and expense.
Video recording of bariatric surgery [9] and left colectomy laparoscopic procedures [10] linked adverse patient outcomes of complications, re-operation and death to differences in technical skills using OSATS evaluations. Video recording using IPS evaluation showed low IPS can predict which surgeons among trainees, practicing surgeons and experts will make critical errors when performing vascular control procedures [5, 8]. Video recording of open surgical procedures in conjunction with hand motion analysis has potential as an unbiased and cost-effective alternative to OSATS, PBA and IPS evaluation of surgeon technical performance. Such video assessment would assist teaching, enable timing of procedural steps, allowing technical skill evaluations to be integrated into residency training. Trainee surgeons could obtain immediate feedback to improve procedural skills and minimize the reinforcement of errors [11]. The elements of manual dexterity on which surgical skill depends have been increasingly well documented over the last decade and are related to levels of experience [12,13,14,15,16,17,18,19,20,21,22,23,24]. However, many studies of surgeon hand motion rely on synthetic models or partial tasks [14, 19, 20] to simplify the analysis or focus on endoscopic/laparoscopic/robotic procedures [15, 17, 18, 22] where the surgeons hands move through a limited range of motions around a fulcrum. Few studies evaluate hand movements occurring during open surgical procedures because these procedures vary widely, requiring assessment methods that allow for freedom of hand and instrument movement. Moreover, ideally, these methods should be sensor-free to avoid interference with hand motion and surgical performance [12, 16, 21]. Adoption of electromagnetic [14] optical tracking [19] and sensors attached to hands to quantitate movements has failed due to complexity or technical difficulty.
The combination of kinematic data collection and analysis, and video surgical gesture-recognition has potential to address these requirements [12, 16, 24]. The entropy of a random variable such as hand motion measures the uncertainty of the chaotic movements. Shannon joint entropy has been used to summarize the systematic information conveyed by bimanual hand movements, using computer vision (CV) algorithms derived from measurement of frequency, direction and speed of movement changes [12, 16, 24]. Entropy should decrease with hand motion efficiency and has been used to analyze hand movements in training models of suturing [12] but not to evaluate open surgical procedures. Our aim in the work reported here was to identify how performance of technical skill during open surgery could be measured unobtrusively by video recording during a surgical procedure on human cadavers. We hypothesized that entropy of surgeon hand motion is congruent with a validated measure of surgeon performance and detects differences between experts, resident surgeons and non-surgeons.
Materials and methods
Axillary artery exposure and proximal control procedures were performed in the State Anatomy Board laboratories at the University of Maryland in Baltimore as part of a study to validate benefits of the Advanced Surgical Skills for Exposure in Trauma course (ASSET) [25]. The cadaver laboratories were equipped with consumer-grade video cameras (Nikon d600) above each table. Shannon joint entropy analysis was used to quantify and summarize bimanual motion [24, 26] captured by 50 frames per second (fps) video using a 50–80 degrees unobstructed field of view that included the incision and operator’s hands.
Variability and ability to discrimination expertise among participants were achieved by testing participants with known differences in skill performing the AA procedure: Two experienced (more than 20 years) attending trauma surgeons, surgical residents (one third-year, one fifth-year) before and immediately after training in AA procedure and 12–18 months later, and two Ph.D. demonstrator/anatomists (anatomically knowledgeable, but inexperienced clinical surgery operators).
Video was reviewed frame-by-frame using VirtualDub version 1.10.4 (http://www.virtualdub.org/). Eight metrics were gathered by trained observers of AA video recordings: (1) total time (skin incision to passage of vessel loop); (2) total idle time [27]; (3) total active time; (4) time from skin incision to division of pectoralis minor; (5) number of times instruments changed; (6) blunt dissection time; (7) sharp dissection time; (8) type and duration of instruments used. Start and stop times, active and idle times associated with blunt dissection, and instrument use were recorded from skin incision to passage of the proximal AA vessel loop. Start and stop times were recorded when a hand or surgical instrument made contact with or left contact with the cadaver. Idle time was determined by summation of the time instruments spent outside the surgical incision. Active time was calculated as the duration of the activity of either or both hands while using instruments or dissecting bluntly. Ratios of active to idle time and sharp to blunt dissection were also calculated. The entropy measures were compared with the individual procedure score for AA procedure collected by co-located trained evaluators present during the AA procedures. Individual procedure score evaluations used a standardized script including checklists and global rating scales as previously described [5,6,7,8]. Measurement of the individual procedure score for AA by analysis of video collected during the evaluation took about five times the duration of the video recorded procedure.
Computer vision algorithm feature extraction and quantification
Sensor-free hand motion feature extraction by computer vision algorithms was accomplished using color-coded surgical gloves: green, dominant hand; orange, non-dominant hand. Left and right hand position for each frame was extracted using computer vision software to detect glove colors (Fig. 1). The change of direction θ was quantified by the angle formed by two consecutive movement directions. Joint entropy was calculated as: \(H\left(X,Y\right)= -\sum p\left(x,y\right) {log}_{2}(x,y)\), where X, Y denote the corresponding measurements of dominant and non-dominant hand motion. Entropy measures (speed = pixels/second; acceleration = change in speed/second; directional change = degrees) were compared between operators. Entropy data could be calculated in near real time with a basic computer (Windows 7 (64 bit) machine with 16 Gigabyte memory, intel i5 core 1.3 GHz).
Results
As shown in Table 1, experts had the shortest times for all time measures, the highest ratio of active to idle times and the fewest instrument changes. Pre-training residents had highest total time, active time, ratio of sharp to blunt dissection and time to pectoralis minor and greatest number of instrument changes. In general, timing and procedural metrics for residents were best immediately after training or when re-evaluated 12 or 18 months later in comparison with pre-training values. Pre-training residents had the longest total time > 1200 s, because neither divided pectoralis minor, a key landmark anterior to the axillary artery, and both failed to expose and encircle the artery with a vessel loop within the 20 min time limit allowed for the procedure. These video analysis metrics were in concordance with individual procedure score changes recorded for the same residents at pre- and post-ASSET and skills-retention evaluations.
Detection of instrument use by different operators
Duration and type of instrument use also differentiated the three operator groups. Anatomists used Debakey forceps much of the time to spread tissue and relied on left- and right-handed blunt dissection. Anatomists also had greater duration for placement of the vessel loop and Army-Navy retractor than the surgeons. Expert surgeons had shorter duration of blunt dissection than residents but spent a greater proportion of their operating time using blunt dissection. Numbers of instrument changes also differentiated experts from other operators (Table 1).
Computer vision entropy
Shannon joint entropy data shown as speed, acceleration and direction were least for experts (7.2/9/7.3) versus anatomists (9.4/9.5/3.3) and surgical residents (8.5/8.6/3.4) and aligned with individual procedure score, timing metrics and with levels of experience and training (Table 1). One left-handed expert surgeon showed greater acceleration and directional entropy change with dominant hand but same speed versus right hand, whereas all the right-handed surgeons showed higher speed with right versus left hands but similar acceleration and directional entropy.
Discussion
Hand motion entropy differed between levels of training and was consistent with video-analysis-derived results and with individual procedure score, a benchmark of competence and a means to identify surgeons at all levels of training who are in need of remedial interventions [28]. The entropy finding conforms to the expectation of the dynamic systems theory of motor development which emphasizes a reduction in variability as part of the learning process that would be expected to differ as a result of training [29]. Our findings are also consistent with cognitive science research on motor learning, which shows higher levels of motor complexity in trainees than experts, because learned motor skills are associated with a decrease in movement complexity [30]. Entropy measures confirmed that lower bimanual entropy (equating to smoothness of hand movements with minimal wasted motion) is a reason why experienced and trained surgeons take less time to perform open surgical procedures.
The computer vision algorithm performed very well. It ‘locked onto’ the operator’s colored gloves and was not distracted by a different colored glove or when both of the operator’s (separately colored) gloves left the operating field and then one reentered. (See attached multimedia video.) However, we were unable to glean sufficient data on fine motor movements—carpal/metacarpal/digital—versus gross movement of the hand to use these analyses to explore in greater detail the basis of gestures associated with particular instruments or particular uses such as cutting, spreading and clamping or suturing gestures. Gesture recognition is captured in computer game users [12], these technologies might have potential for solving the problem of fine-motor analysis and advancing the precision surgical hand motion evaluations. In addition, devices such as Myoband® worn on the forearm can detect muscle movements associated with individual finger movements [31] and may assist trainee feedback. Gesture recognition and finger movements associated with specific instruments and procedural steps could be measured for many different surgical procedures. Frequency and timing of common surgical tasks such as skin incision, cut and spreading, retractor insertion, clamping and suturing could be monitored. Imbedding specialized motion detectors and identifiers into the surgical instruments themselves [32] would facilitate deconstruction and evaluation of the motions associated with a variety of procedures, not just the AA. Video recording with task analysis and hand motion entropy could be incorporated to make objective skill evaluations at all levels of surgical expertise. It could be integrated into routine residency operating-room training, provide experienced surgeons with valuable immediate feedback, quantitate inefficiencies (e.g. idle time, repeated instrument changes) and enable on-site mentored training to improve procedural skills, assist teaching and minimize the reinforcement of errors in procedural steps.
Expert performance
Expert performance was characterized by a minimal amount of idle time and the lowest number of instrument changes during the procedure. Experts’ calculated joint entropy was least among all the operators. The idle time and instrument-change metrics are technical assessments that can easily be targeted for continuing education/skills-improvement for individual surgeons and so may represent a valuable training tool. Although joint entropy will be an unfamiliar concept to many, our data show that overall efficiency and smoothness of motion are captured by this metric, and that it is therefore an important contribution to targeted skills acquisition and maintenance.
Potential benefits for surgical training programs
Both computer programs used for this work are available for free: Shannon entropy (https://biomedical-engineering-online.biomedcentral.com/articles/10.1186/s12938-019-0650-5) and computer vision algorithms (OpenCV Python library, Copyright 2013, Alexander Mordvintsev & Abid K. Revision). High definition video can be collected using consumer level digital cameras costing $1200 or less. As noted above, addition of detectors on instruments [32] could avoid burdensome collection of timing data and enable other benefits such as timing of start and end of surgery and implementation of novel performance assessments. Interval measurements of surgeons performing eligible procedures in a training program could be a means of assessing resident performance and identification of need for individualized remediation. Remediation plans could be tailored to discrete steps of the procedure and range anywhere from reviewing anatomy to repeating individual procedures or tasks in a mentored fashion on a cadaver or a simulator. Use of Shannon entropy with procedural video-analysis could provide objective evaluation of sub-tasks in a procedure such as knot tying and suturing, and assist surgical quality assurance programs or training course evaluations. With routine use in training programs of differentially colored gloves and video recording of the surgeon’s hands and the operative site of specific procedures, computer vision algorithms would be an objective means of evaluating residents. Tools and consent for video recording and analysis in health care settings are summarized elsewhere [33]. These evaluations are inexpensive, might have applicability in both sophisticated surgical programs and in countries where resources and mentors time-availability are limited.
Limitations
Video acquisition requires that the angle and distance of the camera from the operative site be standardized if comparative measures are to be made in the same individuals. The collection of entropy data will not be valid if the hands cannot be visualized (e.g., intra-abdominal procedures) or when the operative field is obscured for any reason (e.g., operating lights). The study experimental design was based on a convenience sample of videos and participants, so that only proof-of-concept examples were obtained from each of the categories of operators. A prospective study confirming data collection and analysis methods and their relation to surgical performance is needed.
Conclusion
In this proof-of-concept study, we show that Shannon joint entropy analysis, number of instrument changes, total time and ratios of idle to active and blunt and sharp dissection times as a proportion of total procedure time, can be used to discriminate usefully between expert and non-expert surgeons and non-clinicians, demonstrating the effects of training on hand motion. Hand motion metrics were congruent with a labor-intensive validated procedure performance score, suggesting that video recording with automated Shannon entropy measures can contribute to the range of modalities available to educators, training course designers, and surgical quality assurance programs and be helpful to surgical trainees by providing feedback of intra-operative technical skills.
References
Jaffer A, Bednarz B, Challacombe B et al (2009) The assessment of surgical competency in the UK. Int J Surg 7:12–15
Grantcharov TP, Bardram L, Funch-Jensen P et al (2002) Assessment of technical surgical skills. Eur J Surg 168:139–144
Martin JA, Regehr G, Reznick R et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278
Mayne A, Wilson L, Kennedy N (2020) The usefulness of procedure-based assessments in postgraduate surgical training within the Intercollegiate Curriculum Programme, aA scoping review. J Surg Educ (in Press). https://doi.org/10.1016/j.jsurg.2020.03.005
Mackenzie CF, Shackelford SA, Tisherman ST et al (2019) Critical errors in infrequently performed trauma procedures after training. Surgery 166:835–843
Mackenzie CF, Garofalo E, Shackelford S et al (2015) Using an Individual Procedure Score before and after the advanced surgical skills exposure for trauma course training to benchmark a hemorrhage-control performance metric. J Surg Educ 72:278–287
Mackenzie CF, Garofalo E, Puche A et al (2017) Performance of vascular exposure and fasciotomy among surgical residents before and after training compared with experts. JAMA Surg 52(6):581–588
Mackenzie CF, Bowyer MW, Henry S et al (2018) Cadaver-based trauma procedural skills training: skills retention 30 months after training among practicing surgeons in comparison to experts and more recently trained residents. J Am Coll Surg 227:270–279
Birkmeyer JD, Finks JF, O’Reilly A et al (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442
Stulberg JJ, Huang R, Kreutzer L et al (2020) Association between surgeon technical skills and patient outcomes. JAMA Surg. https://doi.org/10.1001/jamasurg.2020.3007
Gambadauro P, Margos A (2012) Surgical videos for accident analysis, performance improvement, and complication prevention: time for a surgical black box? Surg Innov 19:76–80
Alvarez-Lopez F, Maina MF, Saigí-Rubió F (2019) Use of commercial off-the-shelf devices for the detection of manual gestures in surgery: systematic literature review. J Med Internet Res 21:e11925
Ahmidi N, Poddar P, Jones JD et al (2015) Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty. Int J Comput Assist Radiol Surg 10:981–991
Datta V, Mackay S, Mandalia M et al (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193:479–485
Gray RJ, Kahol K, Islam G et al (2012) High-fidelity, low-cost, automated method to assess laparoscopic skills objectively. J Surg Educ 69:335–339
Mackenzie CF, Watts D, Patel R et al (2016) Computer-assisted video hand-motion analysis of surgeon technical performance: a preliminary report of methodology and analyses. Human Factors & Ergonomic Society Proceedings, pp 72–76
Watson RA (2014) Hand motion patterns of fundamentals of laparoscopic surgery certified and noncertified surgeons. Am J Surg 207:226–230
Emura M, Tomikawa M, Kumashiro R et al (2014) Analysis of hand motion differentiates expert and novice surgeons. J Surg Res 188:8–13
Watson RA (2012) Computer-aided feedback of surgical knot tying using optical tracking. J Surg Educ 69:306–310
Watson RA (2014) Use of machine learning algorithm to classify expertise: analysis of hand motion patterns during a simulated surgical task. Acad Med 89:1163–1167
Dosis A, Aggarwal R, Bello F et al (2005) Synchronized video and motion analysis for the assessment of procedures in the operating theater. Arch Surg 140:293–299
Dosis A, Bello F, Gillies D et al (2005) Laparoscopic task recognition using Hidden Markov Models. Stud Health Technol Inform 111:115–122
Cao CL, Milgram P (2005) Direction and location are not sufficient for navigating in nonrigid environments: an empirical study in augmented reality. Presence Teleop Virt 16:584–602
Zappella L, Béjar B, Hager GD et al (2013) Surgical gesture classification from video and kinematic data. Med Image Anal 17:732–745
Bowyer MW, Kuhls DA, Haskin D et al (2013) Advanced surgical skills for exposure in trauma (ASSET): the first 25 courses. J Surg Res 183:553–558
Mackay D (2003) Information theory, inference, and learning algorithms. Cambridge University Press, Cambridge, pp 32–34
D’Angelo AL, Rutherford DN, Ray RD et al (2015) Idle time: an underdeveloped performance metric for assessing surgical skill. Am J Surg 209:645–651
Mackenzie CF, Elster E, Bowyer MW et al (2020) Scoping evidence review on training and skills assessment for open emergency surgery. J Surg Educ 77:1211–1226
Stergiou N, Harbourne R, Cavanaugh J (2006) Optimal movement variability: a new theoretical perspective for neurological physical therapy. J Neurol Phy Ther 30:120–129
Taylor JA, Ivry RB (2014) Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcement learning. Prog Brain Res 210:217–253
SrinivasanVB, Islam M, Zheng W (2018) Finger movement classification from myoelectric signals using convolutional. Neural Networks. Conference: 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO). https://doi.org/10.1109/ROBIO.2018.8664807
Kranzfelder M, Schneider A, Feussner H et al (2013) Shock/Sepsis/Trauma/Critical Care: real-time instrument detection in minimally invasive surgery using radiofrequency identification technology. J Surg Res 185:704–710
Mackenzie CF, Xiao Y (2012) Video analysis: an approach for use in healthcare. In: Carayon P (ed) Handbook of human factors and ergonomics in healthcare and patient safety, 2nd edn, Chapter 31. CRC Press, Boca Raton, pp 521–534.
Acknowledgements
Presented in part at the American College of Surgeons 102nd Clinical Congress, Washington D.C. USA October 2016 and the Human Factors and Ergonomics 60th International Annual Meeting, Washington DC, USA, September 2016. Our thanks to the Maryland State Anatomy Board, Maryland Committee on Trauma, the evaluators and participants, and with grateful appreciation of Cadaver Donors who made this study possible.
Funding
U.S. Army Medical Research & Materiel Command Contract Number: W81XWH-13–2-002.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Drs Mackenzie, Yang, Garofalo and Hu hold U.S. Patent Number 10,692,395 “Automated Surgeon Performance Evaluation.” No other author conflicts informed consent was obtained from all individual participants included in the study. University of Maryland School of Medicine Institutional Review Board approved the recruitment and consent process. Cadaver use was approved by the Maryland State Anatomy Board.
Disclaimer
This research and development project was conducted by the University of Maryland, School of Medicine and was made possible by a cooperative agreement which was awarded and administered by the U.S. Army Medical Research & Materiel Command (USAMRMC) and the Telemedicine & Advanced Technology Research Center (TATRC) at Fort Detrick, MD under Contract Number: W81XWH-13–2-0028 (Colin F Mackenzie M.D.). The views, opinions and/or findings contained in this publication are those of the authors and do not necessarily reflect the views of the Department of Defense and should not be construed as an official DoD position, policy or decision unless so designated by other documentation. No official endorsement should be made.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Multimedia video (see separate file) shows the algorithm “lock onto” the surgeon’s hands, despite other hands coming into the field with different colored gloves, the algorithm is not distracted. The computer vision algorithm “re-attaches” onto the surgeon’s hands immediately when both of the operator’s (separately colored) gloves leave the operating field and then one reenters. Supplementary file1 (MP4 18.3 Mb)
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Mackenzie, C.F., Yang, S., Garofalo, E. et al. Enhanced Training Benefits of Video Recording Surgery With Automated Hand Motion Analysis. World J Surg 45, 981–987 (2021). https://doi.org/10.1007/s00268-020-05916-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00268-020-05916-1