Advertisement

Toward a Neural-Symbolic Framework for Automated Workflow Analysis in Surgery

  • Hirenkumar NakawalaEmail author
  • Elena De Momi
  • Roberto Bianchi
  • Michele Catellani
  • Ottavio De Cobelli
  • Pierre Jannin
  • Giancarlo Ferrigno
  • Paolo Fiorini
Conference paper
  • 11 Downloads
Part of the IFMBE Proceedings book series (IFMBE, volume 76)

Abstract

Learning production rules from continuous data streams, e.g. surgical videos, is a challenging problem. To learn production rules, we present a novel framework consisting of deep learning models and inductive logic programming (ILP) system for learning surgical workflow entities that are needed in subsequent surgical tasks, e.g. “what kind of instruments will be needed in the next step?” As a prototypical scenario, we analyzed the Robot-Assisted Partial Nephrectomy (RAPN) workflow. To verify our framework, first consistent and complete rules were learnt from the video annotations which can classify RAPN surgical workflow and temporal sequence at high-granularity e.g. steps. After we found that RAPN workflow is hierarchical, we used combination of learned predicates, presenting workflow hierarchy, to predict the information on the next step followed by a classification of step sequences with deep learning models. The predicted rules on the RAPN workflow was verified by an expert urologist and conforms with the standard workflow of RAPN.

Keywords

Production rules Robot-Assisted Partial Nephrectomy Surgical workflow Deep learning Inductive logic programming 

Notes

Acknowledgement

This research is funded by the European Research Council (ERC) under the European Union’s H2020 research and innovation programme (grant agreement No. 742671 “ARS”). This work was supported by European Union’s Horizon 2020 research and innovation programme (grant agreement No. 732515 “SMARTsurg”).

Conflict of Interest

The authors confirm that there are no known conflicts of interest associated with this publication.

References

  1. 1.
    Nakawala, H., Bianchi, R., Pescatori, L.E., De Cobelli, O., Ferrigno, G., De Momi, E.: “Deep-Onto” network for surgical worflow and context recognition. Int. J. Comput. Assist. Radiol. Surg. 14(4), 685–696 (2019)CrossRefGoogle Scholar
  2. 2.
    Nakawala, H., Ferrigno, G., De Momi, E.: Developement of an intelligent training system for Thoracentesis. Artif. Intell. Med. 84, 50–63 (2018)CrossRefGoogle Scholar
  3. 3.
    Garcez, A., Besold, T.R., de Raedt, L., Földiak, P., Hitzler, P., Icard, T., Kühnberger, K.-U., Lamb, L.C., Miikkulainen, R., Silver, D.L.: Neural-symbolic learning and reasoning: contributions and challenges. In: 2015 AAAI Spring Symposium, Palo Alto, California (2015)Google Scholar
  4. 4.
    Bock, H.-H.: Data Mining Tasks and Methods: Classification: the Goal of Classification. Handbook of Data Mining and Knowledge Discovery. Oxford University Press, New York (2002)Google Scholar
  5. 5.
    Dutra, I., Nassif, H., Page, D., Shavik, J., Strigel, R.M., Wu, Y., Elezaby, M.E., Burnside, E.: Integrating machine learning and physician knowledge to improve the accuracy of breast biopsy. In: AMIA Annual Symposium Proceedings, pp. 349–355 (2011)Google Scholar
  6. 6.
    Burnside, E.S., Davis, J., Costa, V.S., de Castro Dutra, I., Kahn Jr., C.E., Fine, J., Page, D.: Knowledge discovery from structured mammography reports using inductive logic programming. In: AMIA Annual Symposium Proceedings, pp. 96–100 (2005)Google Scholar
  7. 7.
    Evans, R., Grefenstette, E.: Learning explanatory rules from noisy data. J. Artif. Intell. Res. 61(1), 1–64 (2018)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Hu, J.C., Treat, E., Filson, C.P., McLaren, I., Xiong, S., Stepanian, S., Hafez, K.S., Weizer, A.Z., Porter, J.: Technique and outcomes of robot-assisted retroperitoneoscopic partial nephrectomy: a multicenter study. Eur. J. Urol. 66, 542–549 (2014)CrossRefGoogle Scholar
  9. 9.
  10. 10.
    Kipp, M.: Anvil–a generic annotation tool for multimodal dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001)Google Scholar
  11. 11.
    Agrawal, R., Imieliński, T., Swami, A.: Mining association rules between sets of items in large databases. In: Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data - SIGMOD 1993, p. 207 (1993)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Hirenkumar Nakawala
    • 1
    Email author
  • Elena De Momi
    • 2
  • Roberto Bianchi
    • 3
  • Michele Catellani
    • 3
  • Ottavio De Cobelli
    • 3
  • Pierre Jannin
    • 4
  • Giancarlo Ferrigno
    • 2
  • Paolo Fiorini
    • 1
  1. 1.Department of Computer ScienceUniversity of VeronaVeronaItaly
  2. 2.Department of Electronics, Information and Bioengineering (DEIB)Politecnico di MilanoMilanItaly
  3. 3.Department of UrologyEuropean Institute of OncologyMilanItaly
  4. 4.LTSI, INSERM U1099, Universitè de Rennes 1RennesFrance

Personalised recommendations