On-patient see-through augmented reality based on visual SLAM
- 1.6k Downloads
An augmented reality system to visualize a 3D preoperative anatomical model on intra-operative patient is proposed. The hardware requirement is commercial tablet-PC equipped with a camera. Thus, no external tracking device nor artificial landmarks on the patient are required.
We resort to visual SLAM to provide markerless real-time tablet-PC camera location with respect to the patient. The preoperative model is registered with respect to the patient through 4–6 anchor points. The anchors correspond to anatomical references selected on the tablet-PC screen at the beginning of the procedure.
Accurate and real-time preoperative model alignment (approximately 5-mm mean FRE and TRE) was achieved, even when anchors were not visible in the current field of view. The system has been experimentally validated on human volunteers, in vivo pigs and a phantom.
The proposed system can be smoothly integrated into the surgical workflow because it: (1) operates in real time, (2) requires minimal additional hardware only a tablet-PC with camera, (3) is robust to occlusion, (4) requires minimal interaction from the medical staff.
KeywordsAugmented reality Visual SLAM Registration Operating room Surface meshes
This work is supported by the Direccíon General de Investigacíon Centífica y Técnica of Spain under Project RT-SLAM DPI2015-67275-P.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Additionally, all applicable international, national and/or institutional guidelines for the care and use of animals were followed.
Informed consent was obtained from all individual participants included in the study.
Supplementary material 1 (avi 181424 KB)
- 1.Hallet J, Soler L, Diana M, Mutter D, Baumert TF, Habersetzer F, Marescaux J, Pessaux P (2015) Trans-thoracic minimally invasive liverresection guided by augmented reality. J AmColl Surgeons 220(5):e55e60Google Scholar
- 2.Kilgus T, Heim E, Haase S, Prufer S, Muller M, Seitel A, Fangerau M, Wiebe T, Iszatt J, Schlemmer HP, Hornegger J, Yen K, Maier-Hein L (2015) Mobile markerless augmented reality and its application in forensic medicine. IJCARS 10(5):573–586Google Scholar
- 4.Macedo M, Souza A, Giraldi G (2014) High-quality on-patient medical data visualization in a markerless augmented reality environment. SBC J Interact Syst 5(3):41–52Google Scholar
- 8.Muller M, Rassweiler M, Klein J, Seitel A, Gondan M, Baumhauer M, Teber G, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. IJCARS 8(4):663–675Google Scholar
- 10.Schneider A, Baumberger C, Griessen M, Pezold S, Beinemann J, Philipp Jurgens P, Cattin PC (2014) Landmark-based surgical navigation. Clinical image-based procedures. Transl Res Med Imaging 8361:57–64Google Scholar
- 11.Davison AJ (2003) Real-time simultaneous localisation and mapping with a single camera. IEEE Int Conf Comput Vis 2:1403–1410Google Scholar
- 15.Grasa OG, Civera J, Montiel JMM (2009) EKF monocular SLAM 3D modeling. Measuring and augmented reality from endoscope image sequences. In MICCAI, vol 2Google Scholar
- 16.Klein G, Murray D (2007) Parallel tracking and mapping for small AR workspace. In: IEEE and ACM international symposium on mixed and augmented reality (ISMAR), pp 1–10Google Scholar
- 17.Mikhail EM, Bethel JS, McGlone JC (2001) Introduction to modern photogrammetry. Wiley, New YorkGoogle Scholar
- 19.Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: an efficient alternative to SIFT or SURF. In: IEEE international conference on computer vision (ICCV), pp 2564–2571Google Scholar
- 20.Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of IEEE international conference on computer vision, vol 2, pp 1150–1157Google Scholar
- 21.Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. In: Proceedings of 9th European conference on computer vision, pp 430–443Google Scholar
- 22.Shi J, Tomasi C (1994) Good features to track. In: IEEE computer society conference in computer vision and pattern recognition, pp 593–600Google Scholar
- 24.http://ceres-solver.org/. Accessed 4 Apr 2016
- 26.Gálvez-López D, Salas M, Tard JD, Montiel JMM (2015) Real-time monocular object SLAM. J Robots Auton Syst. doi: 10.1016/j.robot.2015.08.009
- 29.https://www.visiblepatient.com/en/. Accessed 4 Apr 2016