Advertisement

International Journal of Computer Vision

, Volume 86, Issue 2–3, pp 211–228 | Cite as

A Multi-Image Shape-from-Shading Framework for Near-Lighting Perspective Endoscopes

  • Chenyu Wu
  • Srinivasa G. Narasimhan
  • Branislav Jaramaz
Article

Abstract

This article formulates a near-lighting shape-from-shading problem with a pinhole camera (perspective projection) and presents a solution to reconstruct the Lambertian surface of bones using a sequence of overlapped endoscopic images, with partial boundaries in each image. First we extend the shape-from-shading problem to deal with perspective projection and near point light sources that are not co-located with the camera center. Secondly we propose a multi-image framework which can align partial shapes obtained from different images in the world coordinates by tracking the endoscope. An iterative closest point (ICP) algorithm is used to improve the matching and recover complete occluding boundaries of the bone. Finally, a complete and consistent shape is obtained by simultaneously re-growing the surface normals and depths in all views. In order to fulfill our shape-from-shading algorithm, we also calibrate both geometry and photometry for an oblique-viewing endoscope that are not well addressed before in the previous literatures. We demonstrate the accuracy of our technique using simulations and experiments with artificial bones.

Keywords

Multi-image Shape-from-shading Near-lighting Perspective projection Calibration Endoscope Bone 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256. CrossRefGoogle Scholar
  2. Clarkson, M. J., Rueckert, D., King, A. P., Edwards, P. J., Hill, D. L. G., & Hawkes, D. J. (1999). Registration of video images to tomographic images by optimising mutual information using texture mapping. In LNCS: Vol. 1679. Proceedings of the second international conference on medical image computing and computer-assisted intervention (MICCAI’99) (pp. 579–588). Berlin: Springer. CrossRefGoogle Scholar
  3. Courteille, F., Crouzil, A., Durou, J.-D., & Gurdjos, P. (2004). Towards shape from shading under realistic photographic conditions. Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04), 2, 277–280. CrossRefGoogle Scholar
  4. Dey, D., Gobbi, D. G., Slomka, P. J., Surry, K. J. M., & Peters, T. M. (2002). Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: Creating stereoscopic panoramas. IEEE Transactions on Medical Imaging, 21(1), 23–30. CrossRefGoogle Scholar
  5. Durou, J.-D., Falcone, M., & Sagona, M. (2008). Numerical methods for shape-from-shading: A new survey with benchmarks. Computer Vision and Image Understanding, 109(1), 22–43. CrossRefGoogle Scholar
  6. Forster, C. H. Q., & Tozzi, C. L. (2000). Toward 3d reconstruction of endoscope images using shape from shading. In Proceedings of the 13th Brazilian symposium on computer graphics and image processing (SIBGRAPHI’00) (pp. 90–96). Google Scholar
  7. Frankot, R. T., & Chellappa, R. (1988). A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4), 439–451. zbMATHCrossRefGoogle Scholar
  8. Fuchs, H., Livingston, M. A., Raskar, R., Colucci, D., Keller, K., State, A., Crawford, J. R., Rademacher, P., Drake, S. H., & Meyer, A. A. (1998). Augmented reality visualization for laparoscopic surgery. In LNCS: Vol. 1496. Proceedings of the first international conference on medical image computing and computer-assisted intervention (MICCAI’98) (pp. 934–943). Berlin: Springer. Google Scholar
  9. Hartley, R., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd edn.). Cambridge: Cambridge University Press. zbMATHGoogle Scholar
  10. Hasegawa, J. K., & Tozzi, C. L. (1996). Shape from shading with perspective projection and camera calibration. Computers and Graphics, 20(3), 351–364. CrossRefGoogle Scholar
  11. Horn, B. K. P., & Brooks, M. J. (1986). The variational approach to shape from shading. Computer Vision, Graphics, and Image Processing (CVGIP’86), 33(2), 174–208. CrossRefGoogle Scholar
  12. Horn, B. K. P., & Brooks, M. J. (1989). Shape from shading. Cambridge: MIT. Google Scholar
  13. Horn, B. K. P., & Sjoberg, R. W. (1979). Calculating the reflectance map. Applied Optics, 18(11), 1770–1779. CrossRefGoogle Scholar
  14. Ikeuchi, K., & Horn, B. K. P. (1981). Numerical shape from shading and occluding boundaries. Artificial Intelligence, 17(1-3), 141–184. CrossRefGoogle Scholar
  15. Iyengar, A. K. S., Sugimoto, H., Smith, D. B., & Sacks, M. S. (2001). Dynamic in vitro quantification of bioprosthetic heart valve leaflet motion using structured light projection. Annals of Biomedical Engineering, 29(11), 963–973. CrossRefGoogle Scholar
  16. Kimmel, R., & Sethian, J. A. (2001). Optimal algorithm for shape from shading and path planning. Journal of Mathematical Imaging and Vision, 14(3), 237–244. zbMATHCrossRefMathSciNetGoogle Scholar
  17. Kozera, R. (1991). Existence and uniqueness in photometric stereo. Applied Maths and Computation, 44(1), 1–103. zbMATHCrossRefMathSciNetGoogle Scholar
  18. Kozera, R. (1992). On shape recovery from two shading patterns. International Journal of Pattern Recognition and Artificial Intelligence, 6(4), 673–698. CrossRefGoogle Scholar
  19. Kozera, R. (1998). An overview of the shape from shading problem. Machine Graphics and Vision, 7(1), 291–312. MathSciNetGoogle Scholar
  20. Kozera, R., & Noakes, L. (2004). Noise reduction in photometric stereo with non-distant light sources. Proceedings of the International Conference on Computer Vision and Graphics (ICCVG’04), 32, 103–110. Google Scholar
  21. Leclerc, Y. G., & Bobick, A. F. (1991). The direct computation of height from shading. In Proceedings of IEEE computer society conference on computer vision and pattern recognition (CVPR’91) (pp. 552–558). Google Scholar
  22. Lee, K. M., & Kuo, C.-C. J. (1994). Shape from shading with perspective projection. Computer Vision, Graphics, and Image Processing: Image Understanding, 59(2), 202–212. CrossRefGoogle Scholar
  23. Lee, K. M., & Kuo, C.-C. J. (1997). Shape from shading with a generalized reflectance map model. Computer Vision and Image Understanding, 67(2), 143–160. CrossRefGoogle Scholar
  24. Litvinov, A., & Schechner, Y. Y. (2005). Addressing radiometric nonidealities: a unified framework. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2, 52–59. CrossRefGoogle Scholar
  25. Mourgues, F., Devernay, F., Malandain, G., & Coste-Manière, E. (2001). 3d reconstruction of the operating field for image overlay in 3d-endoscopic surgery. In: Proceedings of the IEEE and ACM international symposium on augmented reality (ISAR’01) (pp. 191–192). Google Scholar
  26. Okatani, T., & Deguchi, K. (1997). Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center. Computer Vision and Image Understanding, 66(2), 119–131. CrossRefGoogle Scholar
  27. Penna, M. A. (1989). A shape from shading analysis for a single perspective image of a polyhedron. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6), 545–554. CrossRefGoogle Scholar
  28. Poelman, C. J., & Kanade, T. (1997). A paraperspective factorization method for shape and motion recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(3), 206–218. CrossRefGoogle Scholar
  29. Pollefeys, M., Koch, R., & Gool, L. V. (1999). Self-calibration and metric reconstruction inspite of varying and unknown intrinsic camera parameters. International Journal of Computer Vision, 32(1), 7–25. CrossRefGoogle Scholar
  30. Prados, E., & Faugeras, O. (2003). Perspective shape from shading and viscosity solution. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV’03), 2, 826–831. CrossRefGoogle Scholar
  31. Prados, E., & Faugeras, O. (2005). Shape from shading: a well-posed problem? Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2, 870–877. Google Scholar
  32. Samaras, D., & Metaxas, D. N. (1999). Coupled lighting direction and shape estimation from single images. Proceedings of the International Conference on Computer Vision (ICCV’99), 2, 868–874. CrossRefGoogle Scholar
  33. Seshamani, S., Lau, W., & Hager, G. (2006). Real-time endoscopic mosaicking. In LNCS: Vol. 4190. Proceedings of the Ninth International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI’06) (pp. 355–363). Berlin: Springer. Google Scholar
  34. Stoyanov, D., Darzi, A., & Yang, G. Z. (2005). A practical approach towards accurate dense 3d depth recovery for robotic laparoscopic surgery. Computer Aided Surgery, 10(4), 199–208. CrossRefGoogle Scholar
  35. Tankus, A., Sochen, N., & Yeshurun, Y. (2003). A new perspective [on] shape-from-shading. Proceedings of the Ninth IEEE International Conference on Computer Vision (ICCV’03), 2, 862–869. CrossRefGoogle Scholar
  36. Tankus, A., Sochen, N., & Yeshurun, Y. (2004). Reconstruction of medical images by perspective shape-from-shading. Proceedings of the Pattern Recognition, 17th International Conference on (ICPR’04), 3, 778–781. CrossRefGoogle Scholar
  37. Tankus, A., Sochen, N., & Yeshurun, Y. (2005). Shape-from-shading under perspective projection. International Journal of Computer Vision, 63(1), 21–43. CrossRefGoogle Scholar
  38. Tsai, R. Y. (1987). A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-the-shelf tv cameras and lenses. IEEE Journal of Robotics and Automation, 3, 323–344. CrossRefGoogle Scholar
  39. Woodham, R. J. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144. Google Scholar
  40. Yamaguchi, T., Nakamoto, M., Sato, Y., Konishi, K., Hashizume, M., Sugano, N., Yoshikawa, H., & Tamura, S. (2004). Development of a camera model and calibration procedure for oblique-viewing endoscopes. Computer Aided Surgery, 9(5), 203–214. CrossRefGoogle Scholar
  41. Zhang, R., Tsai, P. S., Cryer, J. E., & Shah, M. (1999). Shape from shading: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8), 690–706. CrossRefGoogle Scholar
  42. Zhang, Z. (1998). A flexible new technique for camera calibration (Technical Report MSR-TR-98-71). Microsoft Research. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Chenyu Wu
    • 1
  • Srinivasa G. Narasimhan
    • 1
  • Branislav Jaramaz
    • 1
    • 2
  1. 1.Carnegie Mellon UniversityPittsburghUSA
  2. 2.Institute of Computer Assisted SurgeryPittsburghUSA

Personalised recommendations