Skip to main content
SpringerLink
Log in

Force estimation from OCT volumes using 3D CNNs

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Estimating the interaction forces of instruments and tissue is of interest, particularly to provide haptic feedback during robot-assisted minimally invasive interventions. Different approaches based on external and integrated force sensors have been proposed. These are hampered by friction, sensor size, and sterilizability. We investigate a novel approach to estimate the force vector directly from optical coherence tomography image volumes.

Methods

We introduce a novel Siamese 3D CNN architecture. The network takes an undeformed reference volume and a deformed sample volume as an input and outputs the three components of the force vector. We employ a deep residual architecture with bottlenecks for increased efficiency. We compare the Siamese approach to methods using difference volumes and two-dimensional projections. Data were generated using a robotic setup to obtain ground-truth force vectors for silicon tissue phantoms as well as porcine tissue.

Results

Our method achieves a mean average error of \({7.7 \pm 4.3}\,{\hbox {mN}}\) when estimating the force vector. Our novel Siamese 3D CNN architecture outperforms single-path methods that achieve a mean average error of \({11.59 \pm 6.7}\,{\hbox {mN}}\). Moreover, the use of volume data leads to significantly higher performance compared to processing only surface information which achieves a mean average error of \({24.38 \pm 22.0}\,{\hbox {mN}}\). Based on the tissue dataset, our methods shows good generalization in between different subjects.

Conclusions

We propose a novel image-based force estimation method using optical coherence tomography. We illustrate that capturing the deformation of subsurface structures substantially improves force estimation. Our approach can provide accurate force estimates in surgical setups when using intraoperative optical coherence tomography.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Wilson EB, Bagshahi H, Woodruff VD (2014) Overview of general advantages, limitations, and strategies. In: Robotics in general surgery. Springer, pp 17–22

  2. Kroh M, Chalikonda S (2015) Essentials of robotic surgery. Springer, Berlin

    Book  Google Scholar 

  3. Diana M, Marescaux J (2015) Robotic surgery. Br J Surg 102(2):e15–e28

    Article  PubMed  CAS  Google Scholar 

  4. De Lorenzo D, De Momi E, Dyagilev I, Manganelli R, Formaglio A, Prattichizzo D, Shoham M, Ferrigno G (2011) Force feedback in a piezoelectric linear actuator for neurosurgery. Int J Med Robot Comput Assist Surg 7(3):268–275

    Google Scholar 

  5. Meli L, Pacchierotti C, Prattichizzo D (2017) Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation. Int J Med Robot Comput Assist Surg. https://doi.org/10.1002/rcs.1809

    Article  Google Scholar 

  6. Okamura AM (2009) Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 19(1):102

    Article  PubMed  PubMed Central  Google Scholar 

  7. Pacchierotti C, Meli L, Chinello F, Malvezzi M, Prattichizzo D (2015) Cutaneous haptic feedback to ensure the stability of robotic teleoperation systems. Int J Robot Res 34(14):1773–1787

    Article  Google Scholar 

  8. Bayle B, Joinie-Maurin M, Barbe L, Gangloff J, De Mathelin M (2014) Robot interaction control in medicine and surgery: original results and open problems. In: Computational surgery and dual training. Springer, pp 169–191

  9. Puangmali P, Liu H, Seneviratne LD, Dasgupta P, Althoefer K (2012) Miniature 3-axis distal force sensor for minimally invasive surgical palpation. IEEE/ASME Trans Mechatron 17(4):646–656

    Article  Google Scholar 

  10. Faragasso A, Bimbo J, Noh Y, Jiang A, Sareh S, Liu H, Nanayakkara T, Wurdemann HA, Althoefer K (2014) Novel uniaxial force sensor based on visual information for minimally invasive surgery. In: 2014 IEEE international conference on ICRA, pp 1405–1410

  11. Sokhanvar S, Dargahi J, Najarian S, Arbatani S (2012) Clinical and regulatory challenges for medical devices tactile sensing and displays. Haptic feedback for minimally invasive surgery and robotics

  12. Greminger MA, Nelson BJ (2004) Vision-based force measurement. IEEE Trans Pattern Anal Mach Intell 26(3):290–298

    Article  PubMed  Google Scholar 

  13. Kim J, Janabi-Sharifi F, Kim J (2010) A haptic interaction method using visual information and physically based modeling. IEEE/ASME Trans Mechatron 15(4):636–645

    Article  Google Scholar 

  14. Noohi E, Parastegari S, Žefran M (2014) Using monocular images to estimate interaction forces during minimally invasive surgery. In: 2014 IEEE/RSJ international conference on IROS, pp 4297–4302

  15. Kim W, Seung S, Choi H, Park S, Ko SY, Park JO (2012) Image-based force estimation of deformable tissue using depth map for single-port surgical robot. In: 2012 12th international conference on ICCAS. IEEE, pp 1716–1719

  16. Greminger MA, Nelson BJ (2003) Modeling elastic objects with neural networks for vision-based force measurement. In: 2003 IEEE/RSJ international conference on intelligent robots and systems 2003 (IROS 2003). Proceedings, vol 2. IEEE, pp 1278–1283

  17. Karimirad F, Chauhan S, Shirinzadeh B (2014) Vision-based force measurement using neural networks for biological cell microinjection. J Biomech 47(5):1157–1163

    Article  PubMed  Google Scholar 

  18. Mozaffari A, Behzadipour S, Kohani M (2014) Identifying the tool-tissue force in robotic laparoscopic surgery using neuro-evolutionary fuzzy systems and a synchronous self-learning hyper level supervisor. Appl Soft Comput 14:12–30

    Article  Google Scholar 

  19. Aviles AI, Alsaleh SM, Sobrevilla P, Casals A (2015) Force-feedback sensory substitution using supervised recurrent learning for robotic-assisted surgery. In: 2015 37th annual international conference of the IEEE EMBC, pp 1–4

  20. Aviles AI, Alsaleh SM, Hahn JK, Casals A (2017) Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach. IEEE Trans Haptics 10(3):431–443

    Article  PubMed  Google Scholar 

  21. Otte C, Beringhoff J, Latus S, Antoni ST, Rajput O, Schlaefer A (2016) Towards force sensing based on instrument-tissue interaction. In: 2016 IEEE international conference on MFI. IEEE, pp 180–185

  22. Rivero AIA, Alsaleh SM, Hahn JK, Casals A (2016) Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach. IEEE Transactions on Haptics

  23. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, London

    Google Scholar 

  24. Leal-Taixé L, Canton-Ferrer C, Schindler K (2016) Learning by tracking: Siamese CNN for robust target association. In: Proceedings of the IEEE CVPR workshops, pp 33–40

  25. Zbontar J, LeCun Y (2015) Computing the stereo matching cost with a convolutional neural network. In: Proceedings of the IEEE CVPR, pp 1592–1599

  26. Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T (2015) Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE ICCV, pp 2758–2766

  27. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: ECCV, pp 630–645

  28. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE CVPR, pp 770–778

  29. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks, vol 15. In: Aistats, p 275

  30. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp 448–456

  31. Kingma D, Ba J (2014) Adam: a method for stochastic optimization. In: ICLR

  32. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  33. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467

  34. Borchani H, Varando G, Bielza C, Larrañaga P (2015) A survey on multi-output regression. Wiley Interdiscipl Rev Data Min Knowl Discov 5(5):216–233

    Article  Google Scholar 

  35. Carrasco-Zevallos O, Keller B, Viehland C, Shen L, Waterman G, Todorich B, Shieh C, Hahn P, Farsiu S, Kuo A, Toth C, Izatt J (2016) Live volumetric (4d) visualization and guidance of in vivo human ophthalmic surgery with intraoperative optical coherence tomography. Sci Rep 6(31):689

    Google Scholar 

  36. Aviles AI, Alsaleh S, Sobrevilla P, Casals A (2016) Exploring the effects of dimensionality reduction in deep networks for force estimation in robotic-assisted surgery. In: SPIE medical imaging. International Society for Optics and Photonics, pp 97,861X–97,861X

  37. Potsaid B, Baumann B, Huang D, Barry S, Cable AE, Schuman JS, Duker JS, Fujimoto JG (2010) Ultrahigh speed 1050nm swept source/fourier domain oct retinal and anterior segment imaging at 100,000 to 400,000 axial scans per second. Opt Express 18(19):20029–20048

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  38. Hillmann D, Franke G, Hinkel L, Bonin T, Koch P, Hüttmann G (2013) Off-axis full-field swept-source optical coherence tomography using holographic refocusing. In: Optical coherence tomography and coherence domain optical methods in biomedicine XVII, vol 8571, p 857104. International Society for Optics and Photonics

  39. Ehlers JP, Srivastava SK, Feiler D, Noonan AI, Rollins AM, Tao YK (2014) Integrative advances for oct-guided ophthalmic surgery and intraoperative oct: microscope integration, surgical instrumentation, and heads-up display surgeon feedback. PLoS One 9(8):e105,224

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hamburg University of Technology, Schwarzenbergstraße 95, 21073, Hamburg, Germany

    Nils Gessert, Jens Beringhoff, Christoph Otte & Alexander Schlaefer

Authors
  1. Nils Gessert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jens Beringhoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christoph Otte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander Schlaefer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nils Gessert.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gessert, N., Beringhoff, J., Otte, C. et al. Force estimation from OCT volumes using 3D CNNs. Int J CARS 13, 1073–1082 (2018). https://doi.org/10.1007/s11548-018-1777-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-018-1777-8

Keywords

Search

Navigation