Force classification during robotic interventions through simulation-trained neural networks

  • Andrea MendizabalEmail author
  • Raphael Sznitman
  • Stephane Cotin
Original Article



Intravitreal injection is among the most frequent treatment strategies for chronic ophthalmic diseases. The last decade has seen a serious increase in the number of intravitreal injections, and with it, adverse effects and drawbacks. To tackle these problems, medical assistive devices for robotized injections have been suggested and are projected to enhance delivery mechanisms for a new generation of pharmacological solutions. In this paper, we present a method aimed at improving the safety characteristics of upcoming robotic systems. Our vision-based method uses a combination of 2D OCT data, numerical simulation and machine learning to classify the range of the force applied by an injection needle on the sclera.


We design a neural network to classify force ranges from optical coherence tomography (OCT) images of the sclera directly. To avoid the need for large real data sets, the network is trained on images of simulated deformed sclera. This simulation is based on a finite element method, and the model is parameterized using a Bayesian filter applied to observations of the deformation in OCT images.


We validate our approach on real OCT data collected on five ex vivo porcine eyes using a robotically guided needle. The thorough parameterization of the simulations leads to a very good agreement between the virtually generated samples used to train the network and the real OCT acquisitions. Results show that the applied force range on real data can be predicted with 93% accuracy.


Through a simulation-trained neural network, our approach estimates the force range applied by a robotically guided needle on the sclera based solely on a single OCT slice of the deformed sclera. Being real-time, this solution can be integrated in the control loop of the system, permitting the prompt withdrawal of the needle for safety reasons.


Finite element modeling Bayesian inference Artificial neural networks Force estimation in robotics 



The authors would like to thank Jan Hermann and Tatiana Fountoukidou for the help in the experimental data gathering and Igor Peterlik for his advice and code on Bayesian filtering.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with animals performed by any of the authors.

Informed consent

This articles does not contain patient data.


  1. 1.
    Ullrich F, Michels S, Lehmann D, Pieters RS, Becker M, Nelson BJ (2016) Assistive device for efficient intravitreal injections. Ophthalmic Surg Lasers Imaging Retina 47(8):752–762CrossRefPubMedGoogle Scholar
  2. 2.
    Meenink HCM, Hendrix R, Naus GJL, Beelen MJ, Nijmeijer H, Steinbuch M, van Oosterhout EJGM, de Smet MD (2012) Robot assisted vitreoretinal surgery. In: Gomes P (ed) Medical robotics. Woodhead Publishing, Amsterdam, pp 185–209CrossRefGoogle Scholar
  3. 3.
    Jagtap AD, Riviere CN (2004) Applied force during vitreoretinal microsurgery with handheld instruments. In: The 26th annual international conference of the IEEE engineering in medicine and biology society, vol 1, pp 2771–2773. IEEEGoogle Scholar
  4. 4.
    Weber S, Gavaghan K, Wimmer W, Williamson T, Gerber N, Anso J, Bell B, Feldmann A, Rathgeb C, Matulic M, Stebinger M, Schneider D, Mantokoudis G, Scheidegger O, Wagner F, Kompis M, Caversaccio M (2017) Instrument flight to the inner ear. Sci Robot 2(4):eaal4916 CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Haidegger T, Benyó B, Kovács L, Benyó Z (2009). Force sensing and force control for surgical robots. In: 7th IFAC symposium on modeling and control in biomedical systems, vol, 7, no 1, pp 413–418Google Scholar
  6. 6.
    Haouchine N, Kuang W, Cotin S, Yip MC (2018) Vision-based force feedback estimation for robot-assisted surgery using instrument-constrained biomechanical 3D maps. IEEE Robot Autom Lett 3:2160–2165 CrossRefGoogle Scholar
  7. 7.
    Mura M, Abu-Kheil Y, Ciuti G, Visentini-Scarzanella M, Menciassi A, Dario P, Dias J, Seneviratne L (2016) Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept. J Micro-Bio Robot 11(1–4):35–45CrossRefGoogle Scholar
  8. 8.
    Aviles AI, Marban A, Sobrevilla P, Fernandez J, Casals A (2014) A recurrent neural network approach for 3D vision-based force estimation. In: International conference on image processing theory, tools and applications (IPTA). pp 1–6. IEEEGoogle Scholar
  9. 9.
    Pakhomov D, Premachandran V, Allan M, Azizian M, Navab N (2017) Deep residual learning for instrument segmentation in robotic surgery. arXiv preprint arXiv:1703.08580
  10. 10.
    Aviles AI, Alsaleh S, Sobrevilla P, Casals A (2015) Sensorless force estimation using a neuro-vision-based approach for robotic-assisted surgery. In: 7th International IEEE/EMBS conference on neural engineering (NER), 2015. pp 86–89. IEEEGoogle Scholar
  11. 11.
    Aggarwal V, Asadi H, Gupta M, Lee JJ, Yu D (2018) Covfefe: a computer vision approach for estimating force exertion. arXiv preprint arXiv:1809.09293
  12. 12.
    Mendizabal A, Fountoukidou T, Hermann J, Sznitman R, Cotin S (2018) A combined simulation and machine learning approach for image-based force classification during robotized intravitreal injections. In: International conference on medical image computing and computer-assisted intervention, pp 12–20Google Scholar
  13. 13.
    Moireau P, Chapelle D (2011) Reduced-order Unscented Kalman Filtering with application to parameter identification in large-dimensional systems. ESAIM: Control Optim Calc Var 17(2):380–405Google Scholar
  14. 14.
    Hamilton KE, Pye DC (2008) Young’s modulus in normal corneas and the effect on applanation tonometry. Optom Vis Sci 85(6):445–450CrossRefPubMedGoogle Scholar
  15. 15.
    Bro-Nielsen M, Cotin S (1996) Real-time volumetric deformable models for surgery simulation using finite elements and condensation. In: Computer graphics forum. Blackwell Science Ltd, Edinburgh, vol 15, no 3, pp 57–66Google Scholar
  16. 16.
    Olsen TW, Sanderson S, Feng X, Hubbard WC (2002) Porcine sclera: thickness and surface area. Investig Ophthalmol Vis Sci 43(8):2529–2532Google Scholar
  17. 17.
    Asejczyk-Widlicka M, Pierscionek BK (2008) The elasticity and rigidity of the outer coats of the eye. Br J Ophthalmol 92(10):1415–1418CrossRefPubMedGoogle Scholar
  18. 18.
    Apostolopoulos S, Sznitman R (2017) Efficient OCT volume reconstruction from slitlamp microscopes. IEEE Trans Biomed Eng 64(10):2403–2410CrossRefPubMedGoogle Scholar
  19. 19.
    Balci Y, Basmak H, Kocaturk BK, Sahin A, Ozdamar K (2010) The importance of measuring intraocular pressure using a tonometer in order to estimate the postmortem interval. Am J Forensic Med Pathol 31(2):151–155CrossRefPubMedGoogle Scholar

Copyright information

© CARS 2019

Authors and Affiliations

  1. 1.InriaStrasbourgFrance
  2. 2.ICube University of StrasbourgStrasbourgFrance
  3. 3.University of BernBernSwitzerland

Personalised recommendations