Skip to main content
SpringerLink
Log in

On-patient see-through augmented reality based on visual SLAM

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

An augmented reality system to visualize a 3D preoperative anatomical model on intra-operative patient is proposed. The hardware requirement is commercial tablet-PC equipped with a camera. Thus, no external tracking device nor artificial landmarks on the patient are required.

Methods

We resort to visual SLAM to provide markerless real-time tablet-PC camera location with respect to the patient. The preoperative model is registered with respect to the patient through 4–6 anchor points. The anchors correspond to anatomical references selected on the tablet-PC screen at the beginning of the procedure.

Results

Accurate and real-time preoperative model alignment (approximately 5-mm mean FRE and TRE) was achieved, even when anchors were not visible in the current field of view. The system has been experimentally validated on human volunteers, in vivo pigs and a phantom.

Conclusions

The proposed system can be smoothly integrated into the surgical workflow because it: (1) operates in real time, (2) requires minimal additional hardware only a tablet-PC with camera, (3) is robust to occlusion, (4) requires minimal interaction from the medical staff.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Hallet J, Soler L, Diana M, Mutter D, Baumert TF, Habersetzer F, Marescaux J, Pessaux P (2015) Trans-thoracic minimally invasive liverresection guided by augmented reality. J AmColl Surgeons 220(5):e55e60

    Google Scholar 

  2. Kilgus T, Heim E, Haase S, Prufer S, Muller M, Seitel A, Fangerau M, Wiebe T, Iszatt J, Schlemmer HP, Hornegger J, Yen K, Maier-Hein L (2015) Mobile markerless augmented reality and its application in forensic medicine. IJCARS 10(5):573–586

    Google Scholar 

  3. dos Santos T, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, Maier-Hein L (2014) Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med Image Anal 18(7):1101–1114

    Article  PubMed  Google Scholar 

  4. Macedo M, Souza A, Giraldi G (2014) High-quality on-patient medical data visualization in a markerless augmented reality environment. SBC J Interact Syst 5(3):41–52

    Google Scholar 

  5. Lee J, Huang C, Huang T, Hsieh H, Lee S (2012) Medical augment reality using a markerless registration framework. Int J Expert Syst Appl 39(5):5286–5294

    Article  Google Scholar 

  6. Chen X, Xu L, Wang Y, Wang H, Wang F, Zeng X, Wangb Q, Egger J (2015) Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J Biomed Inform 55:124–131

    Article  PubMed  Google Scholar 

  7. Rassweiler JJ, Müller M, Fangerau M, Klein J, Goezen AS, Pereirac P, Meinzerb HP, Teberd D (2012) iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. Eur Urol 61(3):628–631

    Article  PubMed  Google Scholar 

  8. Muller M, Rassweiler M, Klein J, Seitel A, Gondan M, Baumhauer M, Teber G, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. IJCARS 8(4):663–675

    Google Scholar 

  9. Sun Y, Luebbers H, Agbaje J, Schepers S, Vrielinck L, Lambrichts I, Politis C (2013) Validation of anatomical landmarks-based registration for image-guided surgery: an in-vitro study. J Cranio Maxill Surg 41(6):522–526

    Article  Google Scholar 

  10. Schneider A, Baumberger C, Griessen M, Pezold S, Beinemann J, Philipp Jurgens P, Cattin PC (2014) Landmark-based surgical navigation. Clinical image-based procedures. Transl Res Med Imaging 8361:57–64

    Google Scholar 

  11. Davison AJ (2003) Real-time simultaneous localisation and mapping with a single camera. IEEE Int Conf Comput Vis 2:1403–1410

    Google Scholar 

  12. Civera J, Davison AJ, Montiel JMM (2008) Inverse depth parametrization for monocular SLAM. IEEE Trans Robot 24(5):932–945

    Article  Google Scholar 

  13. Civera J, Grasa OG, Davison AJ, Montiel JMM (2010) 1-Point RANSAC for extended Kalman filtering. Application to real time structure from motion and visual odometry. J Field Robot 27(5):609–631

    Article  Google Scholar 

  14. Grasa OG, Bernal E, Casado S, Gil I, Montiel JMM (2014) Visual SLAM for handheld monocular endoscope. IEEE Trans Med Imaging 33(1):135–146

    Article  PubMed  Google Scholar 

  15. Grasa OG, Civera J, Montiel JMM (2009) EKF monocular SLAM 3D modeling. Measuring and augmented reality from endoscope image sequences. In MICCAI, vol 2

  16. Klein G, Murray D (2007) Parallel tracking and mapping for small AR workspace. In: IEEE and ACM international symposium on mixed and augmented reality (ISMAR), pp 1–10

  17. Mikhail EM, Bethel JS, McGlone JC (2001) Introduction to modern photogrammetry. Wiley, New York

    Google Scholar 

  18. Mur-Artal R, Montiel JMM, Tard JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Robot 31(5):1147–1163

    Article  Google Scholar 

  19. Rublee E, Rabaud V, Konolige K, Bradski G (2011) ORB: an efficient alternative to SIFT or SURF. In: IEEE international conference on computer vision (ICCV), pp 2564–2571

  20. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of IEEE international conference on computer vision, vol 2, pp 1150–1157

  21. Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. In: Proceedings of 9th European conference on computer vision, pp 430–443

  22. Shi J, Tomasi C (1994) Good features to track. In: IEEE computer society conference in computer vision and pattern recognition, pp 593–600

  23. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  24. http://ceres-solver.org/. Accessed 4 Apr 2016

  25. Gao X, Hou X, Tang J, Cheng H (2003) Complete solution classification for the perspective-three-point problem. IEEE Trans Pattern Anal 25(8):930–943

    Article  Google Scholar 

  26. Gálvez-López D, Salas M, Tard JD, Montiel JMM (2015) Real-time monocular object SLAM. J Robots Auton Syst. doi:10.1016/j.robot.2015.08.009

  27. Horn BK (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642

    Article  Google Scholar 

  28. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal 22(11):1330–1334

    Article  Google Scholar 

  29. https://www.visiblepatient.com/en/. Accessed 4 Apr 2016

Download references

Acknowledgments

This work is supported by the Direccíon General de Investigacíon Centífica y Técnica of Spain under Project RT-SLAM DPI2015-67275-P.

Author information

Authors and Affiliations

  1. IRCAD (Institut de Recherche contre les Cancers de l’Appareil Digestif), Strasbourg, France

    Nader Mahmoud, Óscar G. Grasa, Stéphane A. Nicolau, Luc Soler & Jacques Marescaux

  2. ICube (UMR 7357 CNRS), Université de Strasbourg, Strasbourg, France

    Nader Mahmoud & Christophe Doignon

  3. Instituto de Investigación en Ingeniería de Aragón (I3A), Universidad de Zaragoza, Saragossa, Spain

    J. M. M. Montiel

Authors
  1. Nader Mahmoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Óscar G. Grasa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stéphane A. Nicolau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christophe Doignon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luc Soler
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jacques Marescaux
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. J. M. M. Montiel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nader Mahmoud.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Additionally, all applicable international, national and/or institutional guidelines for the care and use of animals were followed.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 181424 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahmoud, N., Grasa, Ó.G., Nicolau, S.A. et al. On-patient see-through augmented reality based on visual SLAM. Int J CARS 12, 1–11 (2017). https://doi.org/10.1007/s11548-016-1444-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-016-1444-x

Keywords

Search

Navigation