Auto-calibration Method for Active 3D Endoscope System Using Silhouette of Pattern Projector

  • Ryo Furukawa
  • Masahito Naito
  • Daisuke Miyazaki
  • Masahi Baba
  • Shinsaku Hiura
  • Yoji Sanomura
  • Shinji Tanaka
  • Hiroshi Kawasaki
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10749)

Abstract

In this paper, we develop an active stereo system for endoscope which requires auto-calibration, because a micro pattern projector is inserted through the instrument channel during an operation and cannot be fixed to the endoscope. For solution, a new auto-calibration technique with full 6-DOF estimation of an active stereo system without any extra devices nor extra pattern projections is proposed. In the technique, the pattern projector itself is simultaneously captured with a target scene by an endoscope camera and the silhouette of the pattern projector is used to conduct 2D-3D matching by using the knowledge of the shape of the projector. In addition, the markers which is included in the projection pattern are extracted and the distances from the closest epipolar lines are calculated as for the cost function. To enhance the robustness of the reconstruction, we also propose a simple high dynamic range (HDR) imaging system for an endoscope by alternating the input power of the pattern projector ON and OFF to blink the pattern so that exposure time will vary with beat frequency, realizing a virtual multi-exposure camera. By applying our auto calibration technique with HDR imaging system, we achieved a robust and accurate reconstruction of tissue in metric 3D under practical operation of the endoscopic system, such as reconstruction of the inside of a real stomach of a pig.

Notes

Acknowledgment

This work was supported in part by JSPS KAKENHI Grant No. 15H02779, 16H02849, MIC SCOPE 171507010 and MSR CORE12.

References

  1. 1.
    Furukawa, R., Masutani, R., Miyazaki, D., Baba, M., Hiura, S., Visentini-Scarzanella, M., Morinaga, H., Kawasaki, H., Sagawa, R.: 2-DOF auto-calibration for a 3D endoscope system based on active stereo. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 7937–7941, August 2015Google Scholar
  2. 2.
    Furukawa, R., Kawasaki, H.: Uncalibrated multiple image stereo system with arbitrarily movable camera and projector for wide range scanning. In: IEEE Conference on 3DIM, pp. 302–309 (2005)Google Scholar
  3. 3.
    Nagakura, T., Michida, T., Hirao, M., Kawahara, K., Yamada, K.: The study of three-dimensional measurement from an endoscopic images with stereo matching method. In: World Automation Congress, WAC 2006, pp. 1–4, July 2006Google Scholar
  4. 4.
    Stoyanov, D., Scarzanella, M.V., Pratt, P., Yang, G.-Z.: Real-time stereo reconstruction in robotically assisted minimally invasive surgery. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6361, pp. 275–282. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15705-9_34 CrossRefGoogle Scholar
  5. 5.
    Visentini-Scarzanella, M., Stoyanov, D., Yang, G.: Metric depth recovery from monocular images using shape-from-shading and specularities. In: ICIP, Orlando, USA, pp. 25–28 (2012)Google Scholar
  6. 6.
    Schmalz, C., Forster, F., Schick, A., Angelopoulou, E.: An endoscopic 3D scanner based on structured light. Med. Image Anal. 16(5), 1063–1072 (2012)CrossRefGoogle Scholar
  7. 7.
    Lin, J., Clancy, N.T., Elson, D.S.: An endoscopic structured light system using multispectral detection. Int. J. Comput. Assist. Radiol. Surg. 10(12), 1941–1950 (2015)CrossRefGoogle Scholar
  8. 8.
    Köhler, T., Haase, S., Bauer, S., Wasza, J., Kilgus, T., Maier-Hein, L., Feußner, H., Hornegger, J.: ToF meets RGB: novel multi-sensor super-resolution for hybrid 3-D endoscopy. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 139–146. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-40811-3_18 CrossRefGoogle Scholar
  9. 9.
    Penne, J., Schaller, C., Engelbrecht, R., Maier-Hein, L., Schmauss, B., Meinzer, H.P., Hornegger, J.: Laparoscopic quantitative 3D endoscopy for image guided surgery. In: Bildverarbeitung für die Medizin, pp. 16–20. Citeseer (2010)Google Scholar
  10. 10.
    Furukawa, R., Aoyama, M., Hiura, S., Aoki, H., Kominami, Y., Sanomura, Y., Yoshida, S., Tanaka, S., Sagawa, R., Kawasaki, H.: Calibration of a 3D endoscopic system based on active stereo method for shape measurement of biological tissues and specimen. In: EMBC, pp. 4991–4994 (2014)Google Scholar
  11. 11.
    Furukawa, R., Sanomura, Y., Tanaka, S., Yoshida, S., Sagawa, R., Visentini-Scarzanella, M., Kawasaki, H.: 3D endoscope system using doe projector. In: The 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2016) (2016)Google Scholar
  12. 12.
    Forsyth, D., Ponce, J.: Computer Vision: A Modern Approach, 2nd edn. Pearson Education Inc., London (2011)Google Scholar
  13. 13.
    Furukawa, R., Kawasaki, H.: Self-calibration of multiple laser planes for 3D scene reconstruction. In: 3DPVT, pp. 200–207 (2006)Google Scholar
  14. 14.
    Furukawa, R., Kawasaki, H.: Laser range scanner based on self-calibration techniques using coplanarities and metric constraints. Comput. Vis. Image Underst. 113(11), 1118–1129 (2009)CrossRefGoogle Scholar
  15. 15.
    Yamazaki, S., Mochimaru, M., Kanade, T.: Simultaneous self-calibration of a projector and a camera using structured light. In: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 60–67. IEEE (2011)Google Scholar
  16. 16.
    Debevec, P.E., Malik, J.: Recovering high dynamic range radiance maps from photographs. In: SIGGRAPH 2008, pp. 1–10. ACM, New York (2008)Google Scholar
  17. 17.
    Kalantari, N.K., Shechtman, E., Barnes, C., Darabi, S., Goldman, D.B., Sen, P.: Patch-based high dynamic range video. ACM Trans. Graph. 32(6), 202–1 (2013)CrossRefGoogle Scholar
  18. 18.
    Eilertsen, G., Mantiuk, R., Unger, J.: A comparative review of tone-mapping algorithms for high dynamic range video. In: Computer Graphics Forum, vol. 36, pp. 565–592. Wiley Online Library (2017)Google Scholar
  19. 19.
    Sagawa, R., Ota, Y., Yagi, Y., Furukawa, R., Asada, N., Kawasaki, H.: Dense 3D reconstruction method using a single pattern for fast moving object. In: ICCV (2009)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Ryo Furukawa
    • 1
  • Masahito Naito
    • 1
  • Daisuke Miyazaki
    • 1
  • Masahi Baba
    • 1
  • Shinsaku Hiura
    • 1
  • Yoji Sanomura
    • 2
  • Shinji Tanaka
    • 2
  • Hiroshi Kawasaki
    • 3
  1. 1.Hiroshima City UniversityHiroshimaJapan
  2. 2.Hiroshima University HospitalHiroshimaJapan
  3. 3.Kyushu UniversityFukuokaJapan

Personalised recommendations