Abstract
Purpose
The treatment of cardiovascular diseases requires complex and challenging navigation of a guidewire and catheter. This often leads to lengthy interventions during which the patient and clinician are exposed to X-ray radiation. Deep reinforcement learning approaches have shown promise in learning this task and may be the key to automating catheter navigation during robotized interventions. Yet, existing training methods show limited capabilities at generalizing to unseen vascular anatomies, requiring to be retrained each time the geometry changes.
Methods
In this paper, we propose a zero-shot learning strategy for three-dimensional autonomous endovascular navigation. Using a very small training set of branching patterns, our reinforcement learning algorithm is able to learn a control that can then be applied to unseen vascular anatomies without retraining.
Results
We demonstrate our method on 4 different vascular systems, with an average success rate of 95% at reaching random targets on these anatomies. Our strategy is also computationally efficient, allowing the training of our controller to be performed in only 2 h.
Conclusion
Our training method proved its ability to navigate unseen geometries with different characteristics, thanks to a nearly shape-invariant observation space.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Al-Ahmad O, Ourak M, Van Roosbroeck J, Vlekken J, Poorten EV (2020) Improved fbg-based shape sensing methods for vascular catheterization treatment. IEEE Rob Autom Lett 5(3):4687–4694. https://doi.org/10.1109/LRA.2020.3003291
Bellman R (1957) A Markovian decision process. J Math Mech 6(5):679–684
van den Berg L, Berkhemer O, Fransen P, Beumer D, Lingsma H, Majoie C, Dippel D, Lugt A, Oostenbrugge R, Zwam W, Roos Y, Dijkgraaf M, Yoo A, Schonewille W, Vos JA, Nederkoorn P, Wermer M, van Walderveen M, Staals J, Koudstaal P (2021) Economic evaluation of endovascular treatment for acute ischemic stroke. Stroke 53:102. https://doi.org/10.1161/strokeaha.121.034599
Bitar I, Grange S, Kotronis P, Benkemoun N (2015) A review on various formulations of displacement based multi-fiber straight timoshenko beam finite elements. In: Proc. CIGOS
Chen F, Liu J, Zhang X, Zhang D, Liao H (2020) Improved 3d catheter shape estimation using ultrasound imaging for endovascular navigation: a further study. IEEE J Biomed Health Inform 24(12):3616–3629. https://doi.org/10.1109/JBHI.2020.3026105
Chi W, Liu J, Rafii-Tari H, Riga C, Bicknell C, Yang GZ (2018) Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization. Int J Comput Assist Radiol Surg 13:855–864. https://doi.org/10.1007/s11548-018-1743-5
Chi W, Dagnino G, Kwok TMY, Nguyen A, Kundrat D, Abdelaziz MEMK, Riga C, Bicknell C, Yang GZ (2020) Collaborative robot-assisted endovascular catheterization with generative adversarial imitation learning. In: 2020 IEEE international conference on robotics and automation (ICRA), pp 2414–2420, https://doi.org/10.1109/ICRA40945.2020.9196912
Faure F, Duriez C, Delingette H, Allard J, Gilles B, Marchesseau S, Talbot H, Courtecuisse H, Bousquet G, Peterlik I, Cotin S (2012) SOFA: a multi-model framework for interactive physical simulation. Comput Assist Surg 11:283–321. https://doi.org/10.1007/8415_2012_125
Haarnoja T, Zhou A, Hartikainen K, Tucker G, Ha S, Tan J, Kumar V, Zhu H, Gupta A, Abbeel P, Levine S (2018) Soft Actor-Critic algorithms and applications. arXiv:1812.05905
Jianu T, Huang B, Abdelaziz M, Vu MN, Fichera S, Lee CY, Berthet-Rayne P, y Baena FR, Nguyen A (2023) Cathsim: an open-source simulator for endovascular intervention. arXiv:2208.01455
Karstensen L, Behr T, Pusch TP, Mathis-Ullrich F, Stallkamp J (2020) Autonomous guidewire navigation in a two dimensional vascular phantom. Curr Direct Biomed Eng 6(1):20200007. https://doi.org/10.1515/cdbme-2020-0007
Karstensen L, Ritter J, Hatzl J, Ernst F, Langejürgen J, Uhl C, Mathis-Ullrich F (2023) Recurrent neural networks for generalization towards the vessel geometry in autonomous endovascular guidewire navigation in the aortic arch. Int J Comput Ass Radiol Surg 18:1735–1744. https://doi.org/10.1007/s11548-023-02938-7
Kirk R, Zhang A, Grefenstette E, Rocktäschel T (2023) A survey of zero-shot generalisation in deep reinforcement learning. J Artif Intell Res 76:201–264. https://doi.org/10.1613/jair.1.14174
Kweon J, Kim K, Lee C, Kwon H, Park J, Song K, Kim YI, Park J, Back I, Roh JH, Moon Y, Choi J, Kim YH (2021) Deep Reinforcement Learning for guidewire navigation in coronary artery phantom. IEEE Access 9:166409–166422. https://doi.org/10.1109/ACCESS.2021.3135277
Meng F, Guo S, Zhou W, Chen Z (2022) Evaluation of an autonomous navigation method for vascular interventional surgery in virtual environment. In: proceeding international conference on mechatronics and automation, pp 1599–1604
Miranda V, Neto AA, Freitas GM, Mozelli LA (2023) Generalization in deep reinforcement learning for robotic navigation by reward shaping. IEEE Trans Ind Elect 11:1–8. https://doi.org/10.1109/tie.2023.3290244
Othonos A (1997) Fiber bragg gratings. Review of scientific instruments 68(12):4309–4341
Patel ST, Haser PB, Bush HL, Kent K (1999) The cost-effectiveness of endovascular repair versus open surgical repair of abdominal aortic aneurysms: a decision analysis model. J Vascul Surg 29(6):958–972. https://doi.org/10.1016/S0741-5214(99)70237-5
Puschel A, Schafmayer C, Groß J (2022) Robot-assisted techniques in vascular and endovascular surgery. Langenbecks Arch Surg 407(5):1789–1795. https://doi.org/10.1007/s00423-022-02465-0
Raffin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22(268):1–8
Tian W, Guo J, Guo S, Fu Q (2023) A DDPG-based method of autonomous catheter navigation in virtual environment. In: proceeding international conference on mechatronics and automation, pp 889–893, https://doi.org/10.1109/ICMA57826.2023.10215710
Wang S, Liu Z, Shu X, Cao Y, Zhang L, Xie L (2022) Study on autonomous delivery of guidewire based on improved yolov5s on vascular model platform. In: 2022 IEEE international conference on robotics and biomimetics (ROBIO), pp 1–6, https://doi.org/10.1109/ROBIO55434.2022.10011829
Funding
This work of the Interdisciplinary Thematic Institute HealthTech, as part of the ITI 2021-2028 program of the University of Strasbourg, CNRS and Inserm, was supported by IdEx Unistra (ANR-10-IDEX-0002) and SFRI (STRAT’US project, ANR-20-SFRI-0012) under the framework of the French Investments for the Future Program.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (mp4 26229 KB)
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Scarponi, V., Duprez, M., Nageotte, F. et al. A zero-shot reinforcement learning strategy for autonomous guidewire navigation. Int J CARS (2024). https://doi.org/10.1007/s11548-024-03092-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11548-024-03092-4