Skip to main content

Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization

International Journal of Computer Assisted Radiology and Surgery volume 11, pages 967–975 (2016)Cite this article

Abstract

Purpose

This work proposes a novel algorithm to register cone-beam computed tomography (CBCT) volumes and 3D optical (RGBD) camera views. The co-registered real-time RGBD camera and CBCT imaging enable a novel augmented reality solution for orthopedic surgeries, which allows arbitrary views using digitally reconstructed radiographs overlaid on the reconstructed patient’s surface without the need to move the C-arm.

Methods

An RGBD camera is rigidly mounted on the C-arm near the detector. We introduce a calibration method based on the simultaneous reconstruction of the surface and the CBCT scan of an object. The transformation between the two coordinate spaces is recovered using Fast Point Feature Histogram descriptors and the Iterative Closest Point algorithm.

Results

Several experiments are performed to assess the repeatability and the accuracy of this method. Target registration error is measured on multiple visual and radio-opaque landmarks to evaluate the accuracy of the registration. Mixed reality visualizations from arbitrary angles are also presented for simulated orthopedic surgeries.

Conclusion

To the best of our knowledge, this is the first calibration method which uses only tomographic and RGBD reconstructions. This means that the method does not impose a particular shape of the phantom. We demonstrate a marker-less calibration of CBCT volumes and 3D depth cameras, achieving reasonable registration accuracy. This design requires a one-time factory calibration, is self-contained, and could be integrated into existing mobile C-arms to provide real-time augmented reality views from arbitrary angles.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. Open-source code available at https://github.com/Nerei/kinfu_remake.

References

  1. Dickerson MT, Scot Drysdale RL, McElfresh SA, Welzl E (1994) Fast greedy triangulation algorithms. In: Proceedings of the tenth annual symposium on computational geometry, SCG ’94. pp 211–220. ACM, New York, NY, USA

  2. Diotte B, Fallavollita P, Wang L, Weidert S, Thaller PH, Euler E, Navab N (2012) Radiation-free drill guidance in interlocking of intramedullary nails. In: Ayache N, Delingette H, Golland P, Mori K (eds) Medical image computing and computer-assisted intervention—MICCAI. Springer, Berlin, Heidelberg, pp 18–25

  3. Foix S, Alenya G, Torras C (2011) Lock-in time-of-flight (ToF) cameras: a survey. Sens J IEEE 11(9):1917–1926

    Article  Google Scholar 

  4. Fuerst B, Fotouhi J, Navab N (2015) Vision-based intraoperative cone-beam ct stitching for non-overlapping volumes. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention—MICCAI 2015. Springer, Cham, pp 387–395

  5. Habert S, Gardiazabal J, Fallavollita P, Navab N (2015) RGBDX: first design and experimental validation of a mirror-based RGBD Xray imaging system. In: International symposium on mixed and augmented reality (ISMAR). Fukuoka

  6. Habert S, Meng M, Kehl W, Wang X, Tombari F, Fallavollita P, Navab N (2015) Augmenting mobile c-arm fluoroscopes via stereo-RGBD sensors for multimodal visualization. In: International symposium on mixed and augmented reality (ISMAR). Fukuoka

  7. Kim CW, Lee YP, Taylor W, Oygar A, Kim WK (2008) Use of navigation-assisted fluoroscopy to decrease radiation exposure during minimally invasive spine surgery. Spine J 8(4):584–590

    Article  PubMed  Google Scholar 

  8. Matthews F, Hoigne DJ, Weiser M, Wanner GA, Regazzoni P, Suhm N, Messmer P (2007) Navigating the fluoroscope’s c-arm back into position: an accurate and practicable solution to cut radiation and optimize intraoperative workflow. J Orthop Trauma 21(10):687–692

    Article  PubMed  Google Scholar 

  9. Merloz P, Troccaz J, Vouaillat H, Vasile C, Tonetti J, Eid A, Plaweski S (2007) Fluoroscopy-based navigation system in spine surgery. Proc Inst Mech Eng Part H J Eng Med 221(7):813–820

    CAS  Article  Google Scholar 

  10. Navab N, Blum T, Wang L, Okur A, Wendler T (2012) First deployments of augmented reality in operating rooms. Computer 7:48–55

    Article  Google Scholar 

  11. Navab N, Heining SM, Traub J (2010) Camera augmented mobile c-arm (camc): calibration, accuracy study, and clinical applications. Med Imaging IEEE Trans 29(7):1412–1423

    Article  Google Scholar 

  12. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohi P, Shotton J, Hodges S, Fitzgibbon A (2011) Kinectfusion: Real-time dense surface mapping and tracking. In: International symposium on mixed and augmented reality (ISMAR), pp 127–136

  13. Nicolau S, Lee P, Wu H, Huang M, Lukang R, Soler L, Marescaux J (2011) Fusion of c-arm x-ray image on video view to reduce radiation exposure and improve orthopedic surgery planning: first in-vivo evaluation. In: 15th Annual conference of the international society for computer aided surgery. Computer Assisted Radiology and Surgery

  14. Pfaff JA, Moore GP (2007) Reducing risk in emergency department wound management. Emerg Med Clin N Am 25(1):189–201

    Article  Google Scholar 

  15. Reaungamornrat S, Otake Y, Uneri A, Schafer S, Mirota D, Nithiananthan S, Stayman JW, Kleinszig G, Khanna AJ, Taylor RH, Siewerdsen JH (2012) An on-board surgical tracking and video augmentation system for c-arm image guidance. Int J Comput Assist Radiol Surg 7(5):647–665

    CAS  Article  PubMed  Google Scholar 

  16. Rusu R, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3D registration. In: Robotics and automation (ICRA), IEEE international conference on, pp 3212–3217

  17. Rusu RB, Cousins S (2011) 3d is here: Point cloud library (pcl). In: Robotics and automation (ICRA), IEEE international conference on, pp 1–4. IEEE

  18. Torr P, Zisserman A (2000) Mlesac: a new robust estimator with application to estimating image geometry. Comput Vision Image Underst 78(1):138–156

    Article  Google Scholar 

  19. van de Kraats EB, van Walsum T, Kendrick L, Noordhoek NJ, Niessen WJ (2006) Accuracy evaluation of direct navigation with an isocentric 3d rotational X-ray system. Med Image Anal 10(2):113–124

    Article  PubMed  Google Scholar 

  20. Wang X, Habert S, Meng M, Wang X, Huang CH, Fallavollita P, Navab N (2015) Rgb-d/c-arm calibration and application in medical augmented reality. In: International symposium on mixed and augmented reality (ISMAR). Fukuoka, Japan

  21. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

Download references

Acknowledgments

The authors want to thank Wolfgang Wein and his team from ImFusion GmbH, Munich, for the opportunity of using the ImFusion Suite, and Gerhard Kleinzig and Sebastian Vogt from SIEMENS for their support and making a SIEMENS ARCADIS Orbic 3D available for this research.

Author information

Author notes

    Affiliations

    1. Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA

      Sing Chun Lee, Bernhard Fuerst, Javad Fotouhi, Marius Fischer & Nassir Navab

    2. Computer Aided Medical Procedures, Technische Universität München, Munich, Germany

      Bernhard Fuerst & Nassir Navab

    3. Trauma Surgery Department, Klinikum Innenstadt, LMU Munich, Munich, Germany

      Marius Fischer

    4. Orthopaedic Trauma, Department of Orthopaedic Surgery, Johns Hopkins Hospital, Baltimore, MD, USA

      Greg Osgood

    Authors
    1. Sing Chun Lee
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Bernhard Fuerst
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Javad Fotouhi
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Marius Fischer
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Greg Osgood
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Nassir Navab
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Sing Chun Lee.

    Ethics declarations

    Conflict of interest

    The authors declare that they have no conflict of interest.

    Ethical approval

    This article does not contain any studies with human participants or animals performed by any of the authors.

    Informed consent

    This article does not contain patient data.

    Additional information

    Grants or other notes.

    S. C. Lee and B. Fuerst contributed equally to the work and should be regarded as joint first authors.

    Rights and permissions

    Reprints and Permissions

    About this article

    Verify currency and authenticity via CrossMark

    Cite this article

    Lee, S.C., Fuerst, B., Fotouhi, J. et al. Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization. Int J CARS 11, 967–975 (2016). https://doi.org/10.1007/s11548-016-1396-1

    Download citation

    • Received:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11548-016-1396-1

    Keywords

    • Augmented reality
    • Cone-beam CT
    • C-arm
    • Intra-operative imaging
    • 3D-3D calibration