Force estimation from OCT volumes using 3D CNNs

  • Nils Gessert
  • Jens Beringhoff
  • Christoph Otte
  • Alexander Schlaefer
Original Article



Estimating the interaction forces of instruments and tissue is of interest, particularly to provide haptic feedback during robot-assisted minimally invasive interventions. Different approaches based on external and integrated force sensors have been proposed. These are hampered by friction, sensor size, and sterilizability. We investigate a novel approach to estimate the force vector directly from optical coherence tomography image volumes.


We introduce a novel Siamese 3D CNN architecture. The network takes an undeformed reference volume and a deformed sample volume as an input and outputs the three components of the force vector. We employ a deep residual architecture with bottlenecks for increased efficiency. We compare the Siamese approach to methods using difference volumes and two-dimensional projections. Data were generated using a robotic setup to obtain ground-truth force vectors for silicon tissue phantoms as well as porcine tissue.


Our method achieves a mean average error of \({7.7 \pm 4.3}\,{\hbox {mN}}\) when estimating the force vector. Our novel Siamese 3D CNN architecture outperforms single-path methods that achieve a mean average error of \({11.59 \pm 6.7}\,{\hbox {mN}}\). Moreover, the use of volume data leads to significantly higher performance compared to processing only surface information which achieves a mean average error of \({24.38 \pm 22.0}\,{\hbox {mN}}\). Based on the tissue dataset, our methods shows good generalization in between different subjects.


We propose a novel image-based force estimation method using optical coherence tomography. We illustrate that capturing the deformation of subsurface structures substantially improves force estimation. Our approach can provide accurate force estimates in surgical setups when using intraoperative optical coherence tomography.


Force estimation OCT 3D CNN Siamese CNN 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.


  1. 1.
    Wilson EB, Bagshahi H, Woodruff VD (2014) Overview of general advantages, limitations, and strategies. In: Robotics in general surgery. Springer, pp 17–22Google Scholar
  2. 2.
    Kroh M, Chalikonda S (2015) Essentials of robotic surgery. Springer, BerlinCrossRefGoogle Scholar
  3. 3.
    Diana M, Marescaux J (2015) Robotic surgery. Br J Surg 102(2):e15–e28CrossRefPubMedGoogle Scholar
  4. 4.
    De Lorenzo D, De Momi E, Dyagilev I, Manganelli R, Formaglio A, Prattichizzo D, Shoham M, Ferrigno G (2011) Force feedback in a piezoelectric linear actuator for neurosurgery. Int J Med Robot Comput Assist Surg 7(3):268–275Google Scholar
  5. 5.
    Meli L, Pacchierotti C, Prattichizzo D (2017) Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation. Int J Med Robot Comput Assist Surg. CrossRefGoogle Scholar
  6. 6.
    Okamura AM (2009) Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 19(1):102CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Pacchierotti C, Meli L, Chinello F, Malvezzi M, Prattichizzo D (2015) Cutaneous haptic feedback to ensure the stability of robotic teleoperation systems. Int J Robot Res 34(14):1773–1787CrossRefGoogle Scholar
  8. 8.
    Bayle B, Joinie-Maurin M, Barbe L, Gangloff J, De Mathelin M (2014) Robot interaction control in medicine and surgery: original results and open problems. In: Computational surgery and dual training. Springer, pp 169–191Google Scholar
  9. 9.
    Puangmali P, Liu H, Seneviratne LD, Dasgupta P, Althoefer K (2012) Miniature 3-axis distal force sensor for minimally invasive surgical palpation. IEEE/ASME Trans Mechatron 17(4):646–656CrossRefGoogle Scholar
  10. 10.
    Faragasso A, Bimbo J, Noh Y, Jiang A, Sareh S, Liu H, Nanayakkara T, Wurdemann HA, Althoefer K (2014) Novel uniaxial force sensor based on visual information for minimally invasive surgery. In: 2014 IEEE international conference on ICRA, pp 1405–1410Google Scholar
  11. 11.
    Sokhanvar S, Dargahi J, Najarian S, Arbatani S (2012) Clinical and regulatory challenges for medical devices tactile sensing and displays. Haptic feedback for minimally invasive surgery and roboticsGoogle Scholar
  12. 12.
    Greminger MA, Nelson BJ (2004) Vision-based force measurement. IEEE Trans Pattern Anal Mach Intell 26(3):290–298CrossRefPubMedGoogle Scholar
  13. 13.
    Kim J, Janabi-Sharifi F, Kim J (2010) A haptic interaction method using visual information and physically based modeling. IEEE/ASME Trans Mechatron 15(4):636–645CrossRefGoogle Scholar
  14. 14.
    Noohi E, Parastegari S, Žefran M (2014) Using monocular images to estimate interaction forces during minimally invasive surgery. In: 2014 IEEE/RSJ international conference on IROS, pp 4297–4302Google Scholar
  15. 15.
    Kim W, Seung S, Choi H, Park S, Ko SY, Park JO (2012) Image-based force estimation of deformable tissue using depth map for single-port surgical robot. In: 2012 12th international conference on ICCAS. IEEE, pp 1716–1719Google Scholar
  16. 16.
    Greminger MA, Nelson BJ (2003) Modeling elastic objects with neural networks for vision-based force measurement. In: 2003 IEEE/RSJ international conference on intelligent robots and systems 2003 (IROS 2003). Proceedings, vol 2. IEEE, pp 1278–1283Google Scholar
  17. 17.
    Karimirad F, Chauhan S, Shirinzadeh B (2014) Vision-based force measurement using neural networks for biological cell microinjection. J Biomech 47(5):1157–1163CrossRefPubMedGoogle Scholar
  18. 18.
    Mozaffari A, Behzadipour S, Kohani M (2014) Identifying the tool-tissue force in robotic laparoscopic surgery using neuro-evolutionary fuzzy systems and a synchronous self-learning hyper level supervisor. Appl Soft Comput 14:12–30CrossRefGoogle Scholar
  19. 19.
    Aviles AI, Alsaleh SM, Sobrevilla P, Casals A (2015) Force-feedback sensory substitution using supervised recurrent learning for robotic-assisted surgery. In: 2015 37th annual international conference of the IEEE EMBC, pp 1–4Google Scholar
  20. 20.
    Aviles AI, Alsaleh SM, Hahn JK, Casals A (2017) Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach. IEEE Trans Haptics 10(3):431–443CrossRefPubMedGoogle Scholar
  21. 21.
    Otte C, Beringhoff J, Latus S, Antoni ST, Rajput O, Schlaefer A (2016) Towards force sensing based on instrument-tissue interaction. In: 2016 IEEE international conference on MFI. IEEE, pp 180–185Google Scholar
  22. 22.
    Rivero AIA, Alsaleh SM, Hahn JK, Casals A (2016) Towards retrieving force feedback in robotic-assisted surgery: a supervised neuro-recurrent-vision approach. IEEE Transactions on HapticsGoogle Scholar
  23. 23.
    Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, LondonGoogle Scholar
  24. 24.
    Leal-Taixé L, Canton-Ferrer C, Schindler K (2016) Learning by tracking: Siamese CNN for robust target association. In: Proceedings of the IEEE CVPR workshops, pp 33–40Google Scholar
  25. 25.
    Zbontar J, LeCun Y (2015) Computing the stereo matching cost with a convolutional neural network. In: Proceedings of the IEEE CVPR, pp 1592–1599Google Scholar
  26. 26.
    Dosovitskiy A, Fischer P, Ilg E, Hausser P, Hazirbas C, Golkov V, van der Smagt P, Cremers D, Brox T (2015) Flownet: Learning optical flow with convolutional networks. In: Proceedings of the IEEE ICCV, pp 2758–2766Google Scholar
  27. 27.
    He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: ECCV, pp 630–645Google Scholar
  28. 28.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE CVPR, pp 770–778Google Scholar
  29. 29.
    Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks, vol 15. In: Aistats, p 275Google Scholar
  30. 30.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp 448–456Google Scholar
  31. 31.
    Kingma D, Ba J (2014) Adam: a method for stochastic optimization. In: ICLRGoogle Scholar
  32. 32.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  33. 33.
    Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467
  34. 34.
    Borchani H, Varando G, Bielza C, Larrañaga P (2015) A survey on multi-output regression. Wiley Interdiscipl Rev Data Min Knowl Discov 5(5):216–233CrossRefGoogle Scholar
  35. 35.
    Carrasco-Zevallos O, Keller B, Viehland C, Shen L, Waterman G, Todorich B, Shieh C, Hahn P, Farsiu S, Kuo A, Toth C, Izatt J (2016) Live volumetric (4d) visualization and guidance of in vivo human ophthalmic surgery with intraoperative optical coherence tomography. Sci Rep 6(31):689Google Scholar
  36. 36.
    Aviles AI, Alsaleh S, Sobrevilla P, Casals A (2016) Exploring the effects of dimensionality reduction in deep networks for force estimation in robotic-assisted surgery. In: SPIE medical imaging. International Society for Optics and Photonics, pp 97,861X–97,861XGoogle Scholar
  37. 37.
    Potsaid B, Baumann B, Huang D, Barry S, Cable AE, Schuman JS, Duker JS, Fujimoto JG (2010) Ultrahigh speed 1050nm swept source/fourier domain oct retinal and anterior segment imaging at 100,000 to 400,000 axial scans per second. Opt Express 18(19):20029–20048CrossRefPubMedPubMedCentralGoogle Scholar
  38. 38.
    Hillmann D, Franke G, Hinkel L, Bonin T, Koch P, Hüttmann G (2013) Off-axis full-field swept-source optical coherence tomography using holographic refocusing. In: Optical coherence tomography and coherence domain optical methods in biomedicine XVII, vol 8571, p 857104. International Society for Optics and PhotonicsGoogle Scholar
  39. 39.
    Ehlers JP, Srivastava SK, Feiler D, Noonan AI, Rollins AM, Tao YK (2014) Integrative advances for oct-guided ophthalmic surgery and intraoperative oct: microscope integration, surgical instrumentation, and heads-up display surgeon feedback. PLoS One 9(8):e105,224CrossRefGoogle Scholar

Copyright information

© CARS 2018

Authors and Affiliations

  1. 1.Hamburg University of TechnologyHamburgGermany

Personalised recommendations