Skip to main content
SpringerLink
Log in

Vision-based online recognition of surgical activities

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Surgical processes are complex entities characterized by expressive models and data. Recognizable activities define each surgical process. The principal limitation of current vision-based recognition methods is inefficiency due to the large amount of information captured during a surgical procedure. To overcome this technical challenge, we introduce a surgical gesture recognition system using temperature-based recognition.

Methods

An infrared thermal camera was combined with a hierarchical temporal memory and was used during surgical procedures. The recordings were analyzed for recognition of surgical activities. The image sequence information acquired included hand temperatures. This datum was analyzed to perform gesture extraction and recognition based on heat differences between the surgeon’s warm hands and the colder background of the environment.

Results

The system was validated by simulating a functional endoscopic sinus surgery, a common type of otolaryngologic surgery. The thermal camera was directed toward the hands of the surgeon while handling different instruments. The system achieved an online recognition accuracy of 96 % with high precision and recall rates of approximately 60 %.

Conclusion

Vision-based recognition methods are the current best practice approaches for monitoring surgical processes. Problems of information overflow and extended recognition times in vision-based approaches were overcome by changing the spectral range to infrared. This change enables the real-time recognition of surgical activities and provides online monitoring information to surgical assistance systems and workflow management systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://www.optris.com/.

  2. http://www.numenta.org/.

References

  1. Archer T, Macario A (2006) The drive for operating room efficiency will increase quality of patient care. Curr Opin Anaesthesiol 19:171–176. doi:10.1097/01.aco.0000192796.02797.82

    Article  PubMed  Google Scholar 

  2. Sutherland J, van den Heuvel W-J (2006) Towards an intelligent hospital environment: adaptive workflow in the OR of the future. Proceedings of the 39th annual Hawaii international conference on system sciences, volume 05. IEEE Computer Society, Washington, DC, USA, p 100b

  3. Cleary K, Kinsella A, Mun SK (2005) OR 2020 workshop report: operating room of the future. Int Congr Ser 1281:832–838. doi:10.1016/j.ics.2005.03.279

    Article  Google Scholar 

  4. Neumuth T, Jannin P, Schlomberg J et al (2011) Analysis of surgical intervention populations using generic surgical process models. Int J Comput Assist Radiol Surg 6:59–71. doi:10.1007/s11548-010-0475-y

    Article  PubMed  PubMed Central  Google Scholar 

  5. Neumuth T (2013) Surgical process modeling: theory, methods, and applications

  6. Neumuth T, Liebmann P, Wiedemann P, Meixensberger J (2012) Surgical workflow management schemata for cataract procedures. Process model-based design and validation of workflow schemata. Methods Inf Med 51:371–382. doi:10.3414/ME11-01-0093

    Article  PubMed  CAS  Google Scholar 

  7. Franke S, Meixensberger J, Neumuth T (2013) Intervention time prediction from surgical low-level tasks. J Biomed Inform 46:152–159. doi:10.1016/j.jbi.2012.10.002

    Article  PubMed  Google Scholar 

  8. Tiwari V, Dexter F, Rothman BS et al (2013) Explanation for the near-constant mean time remaining in surgical cases exceeding their estimated duration, necessary for appropriate display on electronic white boards. Anesth Analg 117:487–493. doi:10.1213/ANE.0b013e31829772e9

    Google Scholar 

  9. Epstein RH, Dexter F (2012) Mediated interruptions of anaesthesia providers using predictions of workload from anaesthesia information management system data. Anaesth Intensive Care 40:803–812

    PubMed  CAS  Google Scholar 

  10. Lin HC, Shafran I, Yuh D, Hager GD (2006) Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. Comput Aided Surg 11:220–230. doi:10.3109/10929080600989189

    Article  PubMed  Google Scholar 

  11. Judkins TN, Oleynikov D, Stergiou N (2009) Objective evaluation of expert and novice performance during robotic surgical training tasks. Surg Endosc 23:590–597. doi:10.1007/s00464-008-9933-9

    Article  PubMed  Google Scholar 

  12. Reiley CE, Lin HC, Varadarajan B et al (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud Health Technol Inform 132:396

    PubMed  Google Scholar 

  13. Vankipuram M, Kahol K, Cohen T, Patel VL (2009) Visualization and analysis of activities in critical care environments. AMIA Annu Symp Proc 2009:662–666

    PubMed  PubMed Central  Google Scholar 

  14. Bouarfa L, Jonker PP, Dankelman J (2009) Surgical context discovery by monitoring low-level activities in the OR

  15. Neumuth T, Meissner C (2012) Online recognition of surgical instruments by information fusion. Int J Comput Assist Radiol Surg 7:297–304. doi:10.1007/s11548-011-0662-5

    Google Scholar 

  16. Meißner C, Neumuth T (2012) RFID-based surgical instrument detection using Hidden Markov models. Biomed Tech (Berl). doi:10.1515/bmt-2012-4047

  17. Kranzfelder M, Zywitza D, Jell T et al (2012) Real-time monitoring for detection of retained surgical sponges and team motion in the surgical operation room using radio-frequency-identification (RFID) technology: a preclinical evaluation. J Surg Res 175:191–198. doi:10.1016/j.jss.2011.03.029

    Article  PubMed  Google Scholar 

  18. Ahmadi S-A, Padoy N, Heining SM et al (2008) Introducing wearable accelerometers in the surgery room for activity detection. 7. Jahrestagung der Deutschen Gesellschaft fuer Computer-und Roboter-Assistierte Chirurgie (CURAC 2008)

  19. Lester J, Choudhury T, Kern N et al (2005) A hybrid discriminative/generative approach for modeling human activities. In: Proceedings of the international joint conference on artificial intelligence (IJCAI), pp 766–772

  20. Klank U, Padoy N, Feussner H, Navab N (2008) Automatic feature generation in endoscopic images. Int J CARS 3:331–339. doi:10.1007/s11548-008-0223-8

    Article  Google Scholar 

  21. Blum T, Feußner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: Jiang T, Navab N, Pluim J, Viergever M (eds) Medical image computing and computer-assisted intervention—MICCAI 2010. Springer, Berlin, pp 400–407

    Chapter  Google Scholar 

  22. Lalys F, Riffaud L, Bouget D, Jannin P (2011) An application-dependent framework for the recognition of high-level surgical tasks in the OR. In: Fichtinger G, Martel A, Peters T (eds) Medical image computing and computer-assisted intervention—MICCAI 2011. Springer, Berlin, pp 331–338

    Chapter  Google Scholar 

  23. Lalys F, Bouget D, Riffaud L, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J Comput Assist Radiol Surg 8:39–49. doi:10.1007/s11548-012-0685-6

    Article  PubMed  Google Scholar 

  24. Haro BB, Zappella L, Vidal R (2012) Surgical gesture classification from video data. In: Ayache N, Delingette H, Golland P, Mori K (eds) Medical image computing and computer-assisted intervention—MICCAI 2012. Springer, Berlin, pp 34–41

    Chapter  Google Scholar 

  25. Appenrodt J, Al-Hamadi A, Michaelis B (2010) Data gathering for gesture recognition systems based on single color-, stereo color-and thermal cameras. Int J Signal Process Image Process Pattern Recognit 3:37–50

    Google Scholar 

  26. Hawkins J, George D (2006) Hierarchical temporal memory: concepts, theory, and terminology

  27. Hawkins J, Blakeslee S (2007) On intelligence. Macmillan, New York

    Google Scholar 

  28. Kapuscinski T (2010) Using hierarchical temporal memory for vision-based hand shape recognition under large variations in hand’s rotation. In: Rutkowski L, Scherer R, Tadeusiewicz R et al (eds) Artificial intelligence and soft computing. Springer, Berlin, pp 272–279

  29. Kapuscinski T, Wysocki M (2009) Using hierarchical temporal memory for recognition of signed polish words. In: Kurzynski M, Wozniak M (eds) Computer recognition systems 3. Springer, Berlin, pp 355–362

    Chapter  Google Scholar 

  30. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2:1–127. doi:10.1561/2200000006

    Article  Google Scholar 

  31. Bouvrie J, Rosasco L, Poggio T (2009) On invariance in hierarchical models. Adv Neural Inf Process Syst 22:162–170

    Google Scholar 

Download references

Acknowledgments

ICCAS is funded by the German Federal Ministry of Education and Research (BMBF) in the scope of the Unternehmen Region (Grant Number 03Z1LN12) and by the German Ministry of Economics (BMWi) in the scope of the Zentrales Innovationsprogramm Mittelstand (ZIM) (Grant Number KF2036709FO0).

Conflict of interest

The authors declare that they have no conflict of interest.

Author information

Authors and Affiliations

  1. Innovation Center Computer Assisted Surgery, University of Leipzig, Semmelweisstr. 14, Leipzig, 04103, Germany

    Michael Unger, Claire Chalopin & Thomas Neumuth

Authors
  1. Michael Unger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Claire Chalopin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Neumuth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Unger.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Unger, M., Chalopin, C. & Neumuth, T. Vision-based online recognition of surgical activities. Int J CARS 9, 979–986 (2014). https://doi.org/10.1007/s11548-014-0994-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-014-0994-z

Keywords

Search

Navigation