Skip to main content

Precise 3D/2D calibration between a RGB-D sensor and a C-arm fluoroscope

International Journal of Computer Assisted Radiology and Surgery volume 11, pages 1385–1395 (2016)Cite this article

Abstract

Purpose

Calibration and registration are the first steps for augmented reality and mixed reality applications. In the medical field, the calibration between an RGB-D camera and a C-arm fluoroscope is a new topic which introduces challenges.

Method

A convenient and efficient calibration phantom is designed by combining the traditional calibration object of X-ray images with a checkerboard plane. After the localization of the 2D marker points in the X-ray images and the corresponding 3D points from the RGB-D images, we calculate the projection matrix from the RGB-D sensor coordinates to the X-ray, instead of estimating the extrinsic and intrinsic parameters simultaneously.

Validation

In order to evaluate the effect of every step of our calibration process, we performed five experiments by combining different steps leading to the calibration. We also compared our calibration method to Tsai’s method to evaluate the advancement of our solution. At last, we simulated the process of estimating the rotation movement of the RGB-D camera using MATLAB and demonstrate that calculating the projection matrix can reduce the angle error of the rotation.

Results

A RMS reprojection error of 0.5 mm is achieved using our calibration method which is promising for surgical applications. Our calibration method is more accurate when compared to Tsai’s method. Lastly, the simulation result shows that using a projection matrix has a lower error than using intrinsic and extrinsic parameters in the rotation estimation.

Conclusions

We designed and evaluated a 3D/2D calibration method for the combination of a RGB-D camera and a C-arm fluoroscope.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Kersten-Oertel M, Jannin P, Collins DL (2012) DVV: a taxonomy for mixed reality visualization in image guided surgery. IEEE Trans Vis Comput Graph 18(2):332–352

    Article  PubMed  Google Scholar 

  2. Navab N, Blum T, Wang L, Okur A, Wendler T (2012) First deployments of augmented reality in operating rooms. Comput IEEE Comput Soc 7:48–55

    Article  Google Scholar 

  3. Ewers R, Schicho K, Wagner A, Undt G, Seemann R, Figl M, Truppe M (2005) Seven years of clinical experience with teleconsultation in craniomaxillofacial surgery. J Oral Maxillofac Surg 63:1447–1454

    Article  PubMed  Google Scholar 

  4. Johnson LG, Edwards P, Hawkes D (2003) Surface transparency makes stereo overlays unpredictable: the implications for augmented reality. Stud Health Technol Inform 94:131–136

  5. Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L (2013) Real-time range imaging in health care: a survey. In: Grzegorzek M, Theobalt C, Koch R, Kolb A (eds) Time-of-flight and depth imaging. Sensors, algorithms, and applications. Springer, Berlin, Heidelberg, pp 228–254

  6. Wang XL, Stolka PJ, Boctor E, Hager G, Choti M (2012) The Kinect as an interventional tracking system. In: David R. Holmes III, Kenneth H. Wong (eds) SPIE medical imaging 2012: image-guided procedures, robotic interventions and modeling, vol 8316. SPIE, San Diego, California, USA, pp 83160U-83160U-6

  7. Kontschieder P, Dorn JF, Morrison C, Corish R, Zikic D, Sellen A, D’Souza M, Kamm CP, Burggraaff J, Tewarie P, Vogel T, Azzarito M, Glocker B, Dahlke P, Polman C, Kappos L, Uitdehaag B, Criminisi A (2014) Quantifying progression of multiple sclerosis via classification of depth videos. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Springer International Publishing, Switzerland, pp 429–437

  8. Alnowami M, Alnwaimi B, Tahavori F, Copland M, Wells K (2012) A quantitative assessment of using the Kinect for Xbox360 for respiratory surface motion tracking. In: David R. Holmes III, Kenneth H. Wong (eds) SPIE medical imaging, international society for optics and photonics, SPIE, San Diego, California, USA, pp 83161T–83161T-10

  9. Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

    Article  Google Scholar 

  10. Gábor B (2015) Camera calibration with opencv.http://www.swarthmore.edu/NatSci/mzucker1/opencv-2.4.10-docs/doc/tutorials/calib3d/camera_calibration/camera_calibration.html#cameracalibrationopencv. Accessed 24 Nov 2015

  11. Yaniv Z, Joskowicz L, Simkin A, Garza-Jinich M, Milgrom C (1998) Fluoroscopic image processing for computer-aided orthopaedic surgery. In: Wells WM, Colchester A, Delp S (eds) Medical image computing and computer-assisted interventation—MICCAI’98, vol 1496. Springer, Berlin, Heidelberg, pp 325–334

  12. Livyatan H, Yaniv Z, Joskowicz L (2002) Robust automatic C-arm calibration for fluoroscopy-based navigation: a practical approach. In: Dohi T, Kikinis R (eds) Medical image computing and computer-assisted intervention–MICCAI 2002, vol 2489. Springer, Berlin, Heidelberg, pp 60–68

  13. Meng C, Zhang J, Zhou F, Wang T (2014) New method for geometric calibration and distortion correction of conventional C-arm. Comput Biol Med 52:49–56

    Article  PubMed  Google Scholar 

  14. Shechter G, Devernay F, Coste-Manière E, McVeigh ER (2002) Temporal tracking of 3D coronary arteries in projection angiograms. In: Medical imaging 2002, international society for optics and photonics. pp 612–623

  15. Shechter G, Shechter B, Resar JR, Beyar R (2005) Prospective motion correction of X-ray images for coronary interventions. IEEE Trans Med Imaging 24(4):441–450

    Article  PubMed  Google Scholar 

  16. Tsai R (1987) A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lens. IEEE J Robot Autom 3:323–344

    Article  Google Scholar 

  17. Jung J, Lee JY, Jeong Y, Kweon IS (2014) Time-of-flight sensor calibration for a color and depth camera pair. IEEE Trans Pattern Anal Mach Intell 37(7):1501–1513

    Article  Google Scholar 

  18. Shime H, Adelsberger R, Kim JD, Rhee SM, Rhee T, Sim JY, Gross M, Kim C (2012) Time-of-flight sensor and color camera calibration for multi-view acquisition. Vis Comput 28:1139–1151

    Article  Google Scholar 

  19. Zhang C, Zhang Z (2014) Calibration between depth and color sensors for commodity depth cameras. In: Shao L, Han J, Kohli P, Zhang Z (eds) Computer vision and machine learning with RGB-D sensors. Springer International Publishing, Switzerland, pp 47–64

  20. Herrera C, Kannala J, Heikkilä J (2012) Joint depth and color camera calibration with distortion correction. IEEE Trans Pattern Anal Mach Intell 34(10):2058–8828

    Article  Google Scholar 

  21. Raposo C, Barreto JP, Nunes U (2013) Fast and accurate calibration of a Kinect sensor. In: International conference on 3D vision—3DV. pp 342–349

  22. Horaud R, Dornaika F (1995) Hand-eye calibration. Int J Robot Res 14(3):195–210

    Article  Google Scholar 

  23. Mitschke MM, Navab N (2000) Recovering projection geometry: how a cheap camera can outperform an expensive stereo system. In: IEEE conference on computer vision and pattern recognition (CVPR), vol. 1. pp 193–200

  24. Navab N, Bani-Hashemi A, Nadar MS, Wiesent K, Durlak P, Brunner T, Barth K, Graumann R (1998) 3D Reconstruction from projection matrices in a C-arm based 3D-angiography system. In: Wells WM, Colchester A, Delp S (eds) Medical image computing and computer assisted interventation—MICCAI’98, vol 1496. Springer, Berlin, Heidelberg, pp 119–129

  25. Yang L, Zhang L, Dong H, Alelaiwi A, Saddik AE (2015) Evaluating and improving the depth accuracy of Kinect for Windows v2. IEEE Sens J 15(8):4275–4285

    Article  Google Scholar 

  26. Kedgley AE, Fox AV, Jenkyn TR (2012) Image intensifier distortion correction for fluoroscopic RSA: the need for independent accuracy assessment. J Appl Clin Med Phys 13(1):197–204

    Google Scholar 

  27. Navab N, Mitschke M (2001) Method and apparatus using a virtual detector for three-dimensional reconstruction from X-ray images. Patent US 6236704; Filing date: June 30, 1999; Issue date: May 22

  28. Huhle B, Schairer T, Jenke P, Strasser W (2008) Robust non-local denosing of colored depth data. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW’08). pp 1–7

  29. Essmaeel K, Gallo L, Damiani E, De Pietro G, Dipanda A (2012) Temporal denoising of Kinect depth data. In: 2012 Eighth international conference on signal image technology and internet based systems (SITIS). pp 47–52

  30. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, Cambridge

    Book  Google Scholar 

  31. Moré JJ (1978) The Levenberg–Marquardt algorithm: implementation and theory. In: Watson GA (ed) Numerical analysis. Springer, Berlin, Heidelberg, pp 105–116

  32. Horn BKP (2000) Tsai’s camera calibration method revisited. http://people.csail.mit.edu/bkph/articles/Tsai_Revisited.pdf

  33. Golub GH, Van Loan CF (1996) Matrix computation, 3rd edn. Johns Hopkins University Press, Baltimore

    Google Scholar 

  34. Horn BKP (2008) Finding the nearest orthonormal matrix. http://people.csail.mit.edu/bkph/articles/Nearest_Orthonormal_Matrix.pdf

  35. Huynh DQ (2009) Metrics for 3D rotation: comparison and analysis. J Math Imaging Vis 35:155–164

    Article  Google Scholar 

  36. Pauly O, Diotte B, Fallavollita P, Weidert S, Euler E, Navab N (2015) Machine learning-based augmented reality for improved surgical scene understanding. Comput Med Imaging Graph 41:55–60

    Article  PubMed  Google Scholar 

  37. Pauly O, Diotte B, Habert S, Weidert S, Euler E, Fallavollita P, Navab N (2014). Relevance-based visualization to improve surgeon perception. In: Stoyanov D, Collins DL, Sakuma I, Abolmaesumi P, Jannin P (eds) Information processing in computer-assisted interventions, vol 8498. Springer International Publishing, Switzerland, pp 178–185

  38. Pauly O, Katouzian A, Eslami A, Fallavollita P, Navab N (2012). Supervised classification for customized intraoperative augmented reality visualization. In: 2012 IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 311–312

  39. Wieczorek M, Aichert A, Fallavollita P, Kutter O, Ahmadi A, Wang L, Navab N (2011) Interactive 3D visualization of a single-view X-ray image. In: Fichtinger G, Martel A, Peters T (eds) Medical image computing and computer-assisted intervention-MICCAI 2011, vol 6891. Springer, Berlin, Heidelberg, pp 73–80

  40. Erat O, Pauly O, Weidert S, Thaller P, Euler E, Mutschler W, et al (2013) How a surgeon becomes superman by visualization of intelligently fused multi-modalities. In: David R. Holmes, Ziv R. Yaniv (eds) SPIE medical imaging. International Society for Optics and Photonics, vol 8671. SPIE, Florida, USA, pp 86710L–86710L-6

  41. Wang L, Fallavollita P, Brand A, Erat O, Weidert S, Thaller PH et al (2012) Intra-op measurement of the mechanical axis deviation: an evaluation study on 19 human cadaver legs. In: Ayache N, Delingette H, Golland P, Mori K (eds) Medical image computing and computer-assisted intervention—MICCAI 2012. Springer, Berlin, Heidelberg, pp 609–616

  42. Diotte B, Fallavollita P, Wang L, Weidert S, Euler E, Thaller P, Navab N (2015) Multi-modal intra-operative navigation during distal locking of intramedullary nails. Med Imaging IEEE Trans 34(2):487–495

    Article  Google Scholar 

  43. Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, et al (2014) The ‘augmented’circles: a video-guided solution for the down-the-beam positioning of IM nail holes. In: Stoyanov D, Collins DL, Sakuma I, Abolmaesumi P, Jannin P (eds) Information processing in computer-assisted interventions, vol 8498. Springer International Publishing, Switzerland, pp 100–107

  44. Londei R, Esposito M, Diotte B, Weidert S, Euler E, Thaller P, et al (2015) Intra-operative augmented reality in distal locking. Int J Comput Assist Radiol Surg 10(9):1395–1403

Download references

Acknowledgments

This work was partly supported by the China Scholarship Council.

Author information

Affiliations

  1. School of Automation Science and Electrical Engineering, Beihang University, Xueyuan Road 37, Haidian District, Beijing, China

    Xiang Wang

  2. Chair for Computer Aided Medical Procedures, Technische Universität München, Boltzmannstraße 3, 85748, Garching bei München, Germany

    Xiang Wang, Severine Habert, Meng Ma, Chun-Hao Huang, Pascal Fallavollita & Nassir Navab

  3. Johns Hopkins University, Baltimore, MD, USA

    Nassir Navab

Authors
  1. Xiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Severine Habert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chun-Hao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pascal Fallavollita
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nassir Navab
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiang Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Statement of informed consent was not applicable since the manuscript does not contain any patient data.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Wang, X., Habert, S., Ma, M. et al. Precise 3D/2D calibration between a RGB-D sensor and a C-arm fluoroscope. Int J CARS 11, 1385–1395 (2016). https://doi.org/10.1007/s11548-015-1347-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-015-1347-2

Keywords

  • 3D/2D calibration
  • C-arm fluoroscope
  • RGB-D sensor
  • Calibration phantom