An Accuracy Certified Augmented Reality System for Therapy Guidance

  • Stéphane Nicolau
  • Xavier Pennec
  • Luc Soler
  • Nicholas Ayache
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3023)


Our purpose is to provide an augmented reality system for Radio-Frequency guidance that could superimpose a 3D model of the liver, its vessels and tumors (reconstructed from CT images) on external video images of the patient. In this paper, we point out that clinical usability not only need the best affordable registration accuracy, but also a certification that the required accuracy is met, since clinical conditions change from one intervention to the other. Beginning by addressing accuracy performances, we show that a 3D/2D registration based on radio-opaque fiducials is more adapted to our application constraints than other methods. Then, we outline a lack in their statistical assumptions which leads us to the derivation of a new extended 3D/2D criterion. Careful validation experiments on real data show that an accuracy of 2 mm can be achieved in clinically relevant conditions, and that our new criterion is up to 9% more accurate, while keeping a computation time compatible with real-time at 20 to 40 Hz.

After the fulfillment of our statistical hypotheses, we turn to safety issues. Propagating the data noise through both our criterion and the classical one, we obtain an explicit formulation of the registration error. As the real conditions do not always fit the theory, it is critical to validate our prediction with real data. Thus, we perform a rigorous incremental validation of each assumption using successively: synthetic data, real video images of a precisely known object, and finally real CT and video images of a soft phantom. Results point out that our error prediction is fully valid in our application range. Eventually, we provide an accurate Augmented Reality guidance system that allows the automatic detection of potentially inaccurate guidance.


Augmented Reality Video Image Compute Tomog Validation Index Registration Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Balter, J.M., Lam, K.L., McGinn, C.J., Lawrence, T.S., Ten Haken, R.K.: Improvement of CT-based treatment-planning models of abdominals targets using static exhale imaging. Int. J. Radiation Oncology Biol. Phys. 41(4), 939–943 (1998)CrossRefGoogle Scholar
  2. 2.
    Soler, L., et al.: Fully automatic anatomical, pathological, and functional segmentation from CT-scans for hepatic surgery. Computer Aided Surgery 6(3) (2001)Google Scholar
  3. 3.
    Dhome, M., et al.: Determination of the attitude of 3D objects from a single perspective view. IEEE Trans. on PAMI 11(12), 1265–1278 (1989)Google Scholar
  4. 4.
    Haralick, R., et al.: Pose estimation from corresponding point data. IEEE Trans. on Systems, Man. and Cybernetics 19(06), 1426–1446 (1989)CrossRefGoogle Scholar
  5. 5.
    Grimson, W., et al.: An automatic registration method for frameless stereotaxy, image-guided surgery and enhanced reality visualization. IEEE TMI 15(2), 129–140 (1996)Google Scholar
  6. 6.
    Fischler, M., Bolles, R.: Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Com. of the ACM 24(6), 381–395 (1981)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Ganapathy, S.: Decomposition of transformation matrices for robot vision. Pattern Recognition Letters 2(6), 401–412 (1984)CrossRefGoogle Scholar
  8. 8.
    Grossmann, E., Santos-Victor, J.: Uncertainty analysis of 3D reconstruction from uncalibrated views. Image and Vision Computing 18, 685–696 (2000)CrossRefGoogle Scholar
  9. 9.
    Hel-Or, Y., Werman, M.: Pose estimation by fusing noisy data of different dimensions. IEEE Trans. on PAMI 17(2), 195–201 (1995)Google Scholar
  10. 10.
    Lu, C., Hager, G.D., Mjolsness, E.: Fast and globally convergent pose estimation from video images. IEEE Trans. on PAMI 22(6), 610–622 (2000)Google Scholar
  11. 11.
    McGahan, J.F., Dodd III, G.D.: Radiofrequency ablation of the liver: Current status. American Journal of Roentgenology 176(1), 3–16 (2001)Google Scholar
  12. 12.
    Nicolau, S., Pennec, X., Soler, L., Ayache, N.: Evaluation of a new 3D/2D registration criterion for liver radio-frequencies guided by augmented reality. In: Ayache, N., Delingette, H. (eds.) IS4TM 2003. LNCS, vol. 2673, pp. 270–283. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  13. 13.
    Pennec, X., Guttmann, C.R.G., Thirion, J.-P.: Feature-based registration of medical images: Estimation and validation of the pose accuracy. In: Wells, W.M., Colchester, A.C.F., Delp, S.L. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 1107–1114. Springer, Heidelberg (1998)Google Scholar
  14. 14.
    Pennec, X., Thirion, J.-P.: A framework for uncertainty and validation of 3D registration methods based on points and frames. IJCV 25(3), 203–229 (1997)CrossRefGoogle Scholar
  15. 15.
    Phong, T., Horaud, R., Tao, P.: Object pose from 2-D to 3-D points and line correspondences. IJCV 15, 225–243 (1995)CrossRefGoogle Scholar
  16. 16.
    Press, W.H., Flannery, B.P., Teukolsky, S.A., Vetterling, W.T.: Numerical Recipices in C. Cambridge Univ. Press, Cambridge (1991)Google Scholar
  17. 17.
    Roche, A., Pennec, X., Malandain, G., Ayache, N.: Rigid registration of 3D ultrasound with MR images: a new approach combining intensity and gradient information. IEEE TMI 20(10), 1038–1049 (2001)Google Scholar
  18. 18.
    Salvi, J., Armangu, X., Batlle, J.: A comparative review of camera calibrating methods with accuracy evaluation. Pattern Recognition 35(7), 1617–1635 (2002)zbMATHCrossRefGoogle Scholar
  19. 19.
    Viola, P., Wells, W.M.: Alignment by maximization of mutual information. International Journal of Computer Vision 24(2), 137–154 (1997)CrossRefGoogle Scholar
  20. 20.
    Wong, J., Sharpe, M., Jaffray, D., Kini, V., Robertson, J., Stromberg, J., Martinez, A.: The use of active breathing control (ABC) to reduce margin for breathing motion. Int. J. Radiation Oncology Biol. Phys. 44(4), 911–919 (1999)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Stéphane Nicolau
    • 1
    • 2
  • Xavier Pennec
    • 1
  • Luc Soler
    • 2
  • Nicholas Ayache
    • 1
  1. 1.INRIA Sophia, EpidaureSophia-Antipolis Cedex
  2. 2.IRCAD-Hopital CivilStrasbourg Cedex

Personalised recommendations