Surgical Endoscopy

, Volume 33, Issue 11, pp 3732–3740 | Cite as

Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying

  • Karl-Friedrich Kowalewski
  • Carly R. Garrow
  • Mona W. Schmidt
  • Laura Benner
  • Beat P. Müller-Stich
  • Felix NickelEmail author



The most common way of assessing surgical performance is by expert raters to view a surgical task and rate a trainee’s performance. However, there is huge potential for automated skill assessment and workflow analysis using modern technology. The aim of the present study was to evaluate machine learning (ML) algorithms using the data of a Myo armband as a sensor device for skills level assessment and phase detection in laparoscopic training.

Materials and methods

Participants of three experience levels in laparoscopy performed a suturing and knot tying task on silicon models. Experts rated performance using Objective Structured Assessment of Surgical Skills (OSATS). Participants wore Myo armbands (Thalmic Labs™, Ontario, Canada) to record acceleration, angular velocity, orientation, and Euler orientation. ML algorithms (decision forest, neural networks, boosted decision tree) were compared for skill level assessment and phase detection.


28 participants (8 beginner, 10 intermediate, 10 expert) were included, and 99 knots were available for analysis. A neural network regression model had the lowest mean absolute error in predicting OSATS score (3.7 ± 0.6 points, r2 = 0.03 ± 0.81; OSATS min.-max.: 4–37 points). An ensemble of binary-class neural networks yielded the highest accuracy in predicting skill level (beginners: 82.2% correctly identified, intermediate: 3.0%, experts: 79.5%) whereas standard statistical analysis failed to discriminate between skill levels. Phase detection on raw data showed the best results with a multi-class decision jungle (average 16% correctly identified), but improved to 43% average accuracy with two-class boosted decision trees after Dynamic time warping (DTW) application.


Modern machine learning algorithms aid in interpreting complex surgical motion data, even when standard analysis fails. Dynamic time warping offers the potential to process and compare surgical motion data in order to allow automated surgical workflow detection. However, further research is needed to interpret and standardize available data and improve sensor accuracy.


Myo armband Machine learning Neural networks Laparoscopy Surgical education Electromyography Skill assessment Workflow analysis Artificial intelligence Laparoscopic training 


Compliance with ethical standards


Felix Nickel reports receiving travel support for conference participation as well as equipment provided for laparoscopic surgery courses by KARL STORZ, Johnson & Johnson, Intuitive, and Medtronic. Karl-Friedrich Kowalewski, Carly R. Garrow, Mona W. Schmidt, Laura Benner and Beat Müller-Stich have no conflicts of interest or financial ties to disclose.


  1. 1.
    Delaney CP et al (2003) Case-matched comparison of clinical and financial outcome after laparoscopic or open colorectal surgery. Ann Surg 238(1):67PubMedPubMedCentralGoogle Scholar
  2. 2.
    Reza M et al (2006) Systematic review of laparoscopic versus open surgery for colorectal cancer. Br J Surg 93(8):921–928CrossRefGoogle Scholar
  3. 3.
    Nguyen KT et al (2011) Comparative benefits of laparoscopic vs open hepatic resection: a critical appraisal. Arch Surg 146(3):348–356CrossRefGoogle Scholar
  4. 4.
    Shabanzadeh DM, Sørensen LT (2012) Laparoscopic surgery compared with open surgery decreases surgical site infection in obese patients: a systematic review and meta-analysis. Ann Surg 256(6):934–945CrossRefGoogle Scholar
  5. 5.
    Vassiliou MC et al (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190(1):107–113CrossRefGoogle Scholar
  6. 6.
    Miskovic D et al (2012) Learning curve and case selection in laparoscopic colorectal surgery: systematic review and international multicenter analysis of 4852 cases. Dis Colon Rectum 55(12):1300–1310CrossRefGoogle Scholar
  7. 7.
    Nickel F et al (2016) Sequential learning of psychomotor and visuospatial skills for laparoscopic suturing and knot tying-a randomized controlled trial “The shoebox study” DRKS00008668. Langenbecks Arch Surg 401(6):893–901CrossRefGoogle Scholar
  8. 8.
    Martin J et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278CrossRefGoogle Scholar
  9. 9.
    Loukas C (2017) Video content analysis of surgical procedures. Surg Endosc 32:553CrossRefGoogle Scholar
  10. 10.
    Ahmadi S-A et al (2006) Recovery of surgical workflow without explicit models. In: International Conference on medical image computing and computer-assisted intervention. Springer, New YorkGoogle Scholar
  11. 11.
    Bardram JE et al (2011) Phase recognition during surgical procedures using embedded and body-worn sensors. IT University of Copenhagen, CopenhagenCrossRefGoogle Scholar
  12. 12.
    Padoy N et al (2007) A boosted segmentation method for surgical workflow analysis. In: International Conference on medical image computing and computer-assisted intervention. Springer, New YorkGoogle Scholar
  13. 13.
    Katic D et al (2016) Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy. Int J Comput Assist Radiol Surg 11(6):881–888CrossRefGoogle Scholar
  14. 14.
    Rosen J et al (2001) Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans Biomed Eng 48(5):579–591CrossRefGoogle Scholar
  15. 15.
    Reiley CE et al (2011) Review of methods for objective surgical skill evaluation. Surg Endosc 25(2):356–366CrossRefGoogle Scholar
  16. 16.
    Spangenberg N et al (2017) Method for intra-surgical phase detection by using real-time medical device data. In: IEEE 30th International Symposium on computer-based medical systemsGoogle Scholar
  17. 17.
    Ganni S et al (2018) A software-based tool for video motion tracking in the surgical skills assessment landscape. Surg Endosc 32(6):2994CrossRefGoogle Scholar
  18. 18.
    Lin P-J, Chen HY (2018) Design and implement of a rehabilitation system with surface electromyography technology. In: 2018 IEEE International Conference on applied system invention (ICASI). IEEEGoogle Scholar
  19. 19.
    Ryser F et al (2017) Fully embedded myoelectric control for a wearable robotic hand orthosis. IEEE Int Conf Rehabil Robot 2017:615–621PubMedGoogle Scholar
  20. 20.
    Sathiyanarayanan M, Raja S (2016) Myo armband for physiotherapy healtchare: a case study using gesture recognition applicationGoogle Scholar
  21. 21.
    Kutafina E et al (2016) Wearable sensors for eLearning of manual tasks: using forearm EMG in hand hygiene training. Sensors (Basel) 16(8):1221CrossRefGoogle Scholar
  22. 22.
    Jimenez DA et al (2016) Human-computer interaction for image guided surgery systems using physiological signals: application to deep brain stimulation surgery. In VII Latin American Congress on Biomedical Engineering CLAIB 2016, Bucaramanga, Santander, Colombia, October 26th–28th, 2017. SpringerGoogle Scholar
  23. 23.
    Sanchez-Margallo FM et al (2017) Use of natural user interfaces for image navigation during laparoscopic surgery: initial experience. Minim Invasive Ther Allied Technol 26(5):253–261CrossRefGoogle Scholar
  24. 24.
    Romero P et al (2014) Intracorporal suturing—driving license necessary? J Pediatr Surg 49(7):1138–1141CrossRefGoogle Scholar
  25. 25.
    Munz Y et al (2007) Curriculum-based solo virtual reality training for laparoscopic intracorporeal knot tying: objective assessment of the transfer of skill from virtual reality to reality. Am J Surg 193(6):774–783CrossRefGoogle Scholar
  26. 26.
    Kowalewski K-F et al (2016) Development and validation of a sensor- and expert model-based training system for laparoscopic surgery: the iSurgeon. Surg Endosc 31:2155CrossRefGoogle Scholar
  27. 27.
    Chang OH et al (2015) Developing an objective structured assessment of technical skills for laparoscopic suturing and intracorporeal knot tying. J Surg Educ 73:258CrossRefGoogle Scholar
  28. 28.
    Brown JD et al (2017) Using contact forces and robot arm accelerations to automatically rate surgeon skill at peg transfer. IEEE Trans Biomed Eng 64(9):2263–2275CrossRefGoogle Scholar
  29. 29.
    Sakoe H, Chiba S (1978) Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans ASSP 26(1):43CrossRefGoogle Scholar
  30. 30.
    Wang Q (2013) Dynamic Time Warping (DTW). MathWorks file exchange.
  31. 31.
    Ahmidi N et al (2010) Surgical task and skill classification from eye tracking and tool motion in minimally invasive surgery. Med Image Comput Comput Assist Interv 13(Pt 3):295–302PubMedGoogle Scholar
  32. 32.
    Fard MJ et al (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot 14(1):e1850CrossRefGoogle Scholar
  33. 33.
    Rosen J et al (2001) Objective laparoscopic skills assessments of surgical residents using Hidden Markov Models based on haptic information and tool/tissue interactions. Stud Health Technol Inform 81:417–423PubMedGoogle Scholar
  34. 34.
    Oropesa I et al (2013) EVA: laparoscopic instrument tracking based on endoscopic video analysis for psychomotor skills assessment. Surg Endosc 27(3):1029–1039CrossRefGoogle Scholar
  35. 35.
    Murdoch TB, Detsky AS (2013) The inevitable application of big data to health care. Jama 309(13):1351–1352CrossRefGoogle Scholar
  36. 36.
    Raghupathi W, Raghupathi V (2014) Big data analytics in healthcare: promise and potential. Health Inf Sci Syst 2(1):3CrossRefGoogle Scholar
  37. 37.
    Kenngott HG et al (2016) Intelligent operating room suite: from passive medical devices to the self-thinking cognitive surgical assistant. Chirurg 87(12):1033–1038CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of General, Visceral, and Transplantation SurgeryUniversity of HeidelbergHeidelbergGermany
  2. 2.Department of Medical Biometry and InformaticsUniversity of HeidelbergHeidelbergGermany

Personalised recommendations