Bridging the gap between formal and experience-based knowledge for context-aware laparoscopy

  • Darko KatićEmail author
  • Jürgen Schuck
  • Anna-Laura Wekerle
  • Hannes Kenngott
  • Beat Peter Müller-Stich
  • Rüdiger Dillmann
  • Stefanie Speidel
Original Article



Computer assistance is increasingly common in surgery. However, the amount of information is bound to overload processing abilities of surgeons. We propose methods to recognize the current phase of a surgery for context-aware information filtering. The purpose is to select the most suitable subset of information for surgical situations which require special assistance.


We combine formal knowledge, represented by an ontology, and experience-based knowledge, represented by training samples, to recognize phases. For this purpose, we have developed two different methods. Firstly, we use formal knowledge about possible phase transitions to create a composition of random forests. Secondly, we propose a method based on cultural optimization to infer formal rules from experience to recognize phases.


The proposed methods are compared with a purely formal knowledge-based approach using rules and a purely experience-based one using regular random forests. The comparative evaluation on laparoscopic pancreas resections and adrenalectomies employs a consistent set of quality criteria on clean and noisy input. The rule-based approaches proved best with noisefree data. The random forest-based ones were more robust in the presence of noise.


Formal and experience-based knowledge can be successfully combined for robust phase recognition.


Context-awareness Cognition-guided assistance Ontology 



The present research was supported by the “SFB TRR 125” funded by the DFG the ESF of Baden-Wuerttemberg.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. For this type of study formal consent was not required since only secondary data, namely performed annotations without patient identifiers, were used.


  1. 1.
    Skagestad P (1993) Thinking with machines: intelligence augmentation, evolutionary epistemology, and semiotic. J Soc Evol Syst 16(2):157–180CrossRefGoogle Scholar
  2. 2.
    Woods DD, Patterson ES, Roth EM (2002) Can we ever escape from data overload? A cognitive systems diagnosis. Cogn Technol Work 4(1):22–36CrossRefGoogle Scholar
  3. 3.
    Dixon B, Daly MJ, Chan H, Vescan AD, Witterick IJ, Irish JC (2013) Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg Endosc 27(2):454–461CrossRefPubMedGoogle Scholar
  4. 4.
    Joyce JP, Lapinsky GW (1983) A history and overview of the safety parameter display system concept. IEEE Trans Nucl Sci 30(1):744–749CrossRefGoogle Scholar
  5. 5.
    Kranzfelder M, Staub C, Fiolka A, Schneider A, Gillen S, Wilhelm D, Friess H, Knoll A, Feussner H (2013) Toward increased autonomy in the surgical OR: needs, requests, and expectations. Surg Endosc 27(5):1681–1688CrossRefPubMedGoogle Scholar
  6. 6.
    Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Edwards PE, Jannin P, Peters TM, Holmes DR, Robb RA (2013) On mixed reality environments for minimally invasive therapy guidance: systems architecture, successes and challenges in their implementation from laboratory to clinic. Comp Med Imaging Graph 37(2):83–97CrossRefGoogle Scholar
  7. 7.
    Lalys F, Jannin P (2014) Surgical process modelling: a review. Int J Comput Assist Radiol Surg 9(3):495–511. doi: 10.1007/s11548-013-0940-5
  8. 8.
    Blum T, Feussner H, Navab N (2010) Modeling and segmentation of surgical workflow from laparoscopic video. In: Jiang T, Navab N, Pluim JPW, Viergever MA (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2010, vol 6363. Springer, Berlin, pp 400–407Google Scholar
  9. 9.
    Suzuki T, Sakurai Y, Yoshimitsu K, Nambu K, Muragaki Y, Iseki H (2010) Intraoperative multichannel audio–visual information recording and automatic surgical phase and incident detection. In: IEEE EMBS, pp 1190–1193Google Scholar
  10. 10.
    Blum T, Padoy N, Feussner H, Navab N (2008) Workflow mining for visualization and analysis of surgeries. Int J Comput Assist Radiol Surg 3(5):379–386CrossRefGoogle Scholar
  11. 11.
    Bouarfa L, Jonker PP, Dankelman J (2010) Discovery of high-level tasks in the operating room. J Biomed Inform 44(3):455–462CrossRefPubMedGoogle Scholar
  12. 12.
    Ahmadi A, Sielhorst T, Stauder R, Horn M, Feussner H, Navab N (2007) Recovery of surgical workflow without explicit models. In: Proceedings of the MICCAI, pp 420–428Google Scholar
  13. 13.
    Stauder R, Okur A, Peter L, Schneider A, Kranzfelder M, Feussner H, Navab N (2014) Random forests for phase detection in surgical workflow analysis. In: Proceedings of the IPCAIGoogle Scholar
  14. 14.
    Reiley CE, Lin HC, Varadarajan B, Vagvolgyi B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud Health Technol Inform 132:396–401Google Scholar
  15. 15.
    Lalys F, Riffaud L, Morandi X, Jannin P (2011) Surgical phases detection from microscope videos by combining SVM and HMM medical computer vision. Recognition techniques and applications in medical imagingGoogle Scholar
  16. 16.
    Padoy N, Blum T, Ahmadi SA, Feussner H, Berger MO, Navab N (2010) Statistical modeling and recognition of surgical workflow. Med Image Anal 16(3):632–641CrossRefPubMedGoogle Scholar
  17. 17.
    Neumuth T, Jannin P, Schlomberg J, Meixensberger J, Wiedemann P, Burgert O (2011) Analysis of surgical intervention populations using generic surgical process models. Int J Comput Assist Radiol Surg 6(1):59–71CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Burgert O, Neumuth T, Lempp F, Mudunuri R, Meixensberger J, Strauss G, Dietz A, Jannin P, Lemke HU (2006) Linking top-level ontologies and surgical workflows. Int J Comput Assist Radiol Surg 1(1):437–438Google Scholar
  19. 19.
    Jannin P, Morandi X (2007) Surgical models for computer-assisted neurosurgery. Neuroimage 37(3):783–791CrossRefPubMedGoogle Scholar
  20. 20.
    Neumuth T, Kaschek B, Neumuth D, Ceschia M, Meixensberger J, Strauss G, Burgert O (2010) An observation support system with an adaptive ontology-driven user interface for the modeling of complex behaviors during surgical interventions. Behav Res Methods 42(4):1049–1058CrossRefPubMedGoogle Scholar
  21. 21.
    Neumuth D, Loebe F, Herre H, Neumuth T (2011) Modeling surgical processes: a four-level translational approach. Artif Intell Med 51(3):147–161CrossRefPubMedGoogle Scholar
  22. 22.
    Lalys F, Bouget B, Riffaud R, Jannin P (2013) Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures. Int J CARS 8:39–49CrossRefGoogle Scholar
  23. 23.
    Katic D, Julliard C, Wekerle AL, Kenngott H, Müller-Stich BP, Dillmann R, Speidel S, Jannin P, Gibaud B (2015) LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition. Int J CARS 10:1427–1434CrossRefGoogle Scholar
  24. 24.
    Katic D, Wekerle AL, Grtner F, Kenngott H, Müller-Stich BP, Dillmann R, Speidel S (2014) Knowledge-driven formalization of laparoscopic surgeries for rule-based intraoperative context-aware assistance. In: IPCAIGoogle Scholar
  25. 25.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefGoogle Scholar
  26. 26.
    Katic D, Wekerle AL, Gaertner F, Kenngott H, Müller-Stich BP, Dillmann E, Speidel S (2013) Ontology-based prediction of surgical events in laparoscopic surgery. In: Proceedings of the SPIEGoogle Scholar
  27. 27.
    Forestier G, Riffaud L, Jannin P (2015) Automatic phase prediction from low-level surgical activities. Int J Comput Assist Radiol Surg 20(6):833–884CrossRefGoogle Scholar
  28. 28.
    Horrocks I, Patel-Schneider PF, Boley H, Tabet S, Grosof B, Dean M (2004) SWRL: a semantic web rule language combining OWL and RuleML. W3C Member Submission. Accessed 25 Mar 2016
  29. 29.
    Ma H, Wang y (2009) Cultural algorithm based on particle swarm optimization for function optimization. In: Fifth international conference on natural computationGoogle Scholar
  30. 30.
    Kennedy J, Eberhart R (1995) Particle swarm optimization. In: IEEE international conference on NNGoogle Scholar
  31. 31.
    Kobti Z, Reynolds R, Kohler T (2004) Agent-based modeling of cultural change in swarm using cultural algorithms. In: SWARMFESTGoogle Scholar
  32. 32.
    Neumuth T, Jannin P, Strauss G, Meixensberger J, Burgert O (2009) Validation of knowledge acquisition for surgical process models. J Am Med Inform Assoc 16(1):72–80Google Scholar
  33. 33.
    Speidel S, Benzko J, Sudra G, Azad P, Müller-Stich BP, Gutt , Dillmann R (2009) Automatic classification of minimally invasive instruments based on endoscopic image sequences. In: SPIE medical imagingGoogle Scholar
  34. 34.
    Yang Y, Calmet J (2005) OntoBayes: an ontology-driven uncertainty model. In: International conference on intelligent agents, web technologies and internet commerceGoogle Scholar

Copyright information

© CARS 2016

Authors and Affiliations

  • Darko Katić
    • 1
    Email author
  • Jürgen Schuck
    • 1
  • Anna-Laura Wekerle
    • 2
  • Hannes Kenngott
    • 2
  • Beat Peter Müller-Stich
    • 2
  • Rüdiger Dillmann
    • 1
  • Stefanie Speidel
    • 1
  1. 1.Institute for Anthropomatics and Robotics (IAR)Karlsruhe Institute of Technology (KIT)KarlsruheGermany
  2. 2.Department of General, Abdominal and Transplantation SurgeryUniversity of HeidelbergHeidelbergGermany

Personalised recommendations