ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy

  • Thomas Köhler
  • Sven Haase
  • Sebastian Bauer
  • Jakob Wasza
  • Thomas Kilgus
  • Lena Maier-Hein
  • Hubertus Feußner
  • Joachim Hornegger
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8149)

Abstract

3-D endoscopy is an evolving field of research with the intention to improve safety and efficiency of minimally invasive surgeries. Time-of-Flight (ToF) imaging allows to acquire range data in real-time and has been engineered into a 3-D endoscope in combination with an RGB sensor (640×480 px) as a hybrid imaging system, recently. However, the ToF sensor suffers from a low spatial resolution (64×48 px) and a poor signal-to-noise ratio. In this paper, we propose a novel multi-frame super-resolution framework to improve range images in a ToF/RGB multi-sensor setup. Our approach exploits high-resolution RGB data to estimate subpixel motion used as a cue for range super-resolution. The underlying non-parametric motion model based on optical flow makes the method applicable to endoscopic scenes with arbitrary endoscope movements. The proposed method was evaluated on synthetic and real images. Our approach improves the peak-signal-to-noise ratio by 1.6 dB and structural similarity by 0.02 compared to single-sensor super-resolution.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Röhl, S., Bodenstedt, S., Suwelack, S., Kenngott, S., Mueller-Stich, P., Dillmann, R., Speidel, S.: Real-time surface reconstruction from stereo endoscopic images for intraoperative registration. In: Proc. SPIE, vol. 7964, p. 796414 (2011)Google Scholar
  2. 2.
    Field, M., Clarke, D., Strup, S., Seales, W.: Stereo endoscopy as a 3-d measurement tool. In: EMBC 2009, pp. 5748–5751 (2009)Google Scholar
  3. 3.
    Schmalz, C., Forster, F., Schick, A., Angelopoulou, E.: An endoscopic 3d scanner based on structured light. Med. Image Anal. 16(5), 1063–1072 (2012)CrossRefGoogle Scholar
  4. 4.
    Penne, J., Höller, K., Stürmer, M., Schrauder, T., Schneider, A., Engelbrecht, R., Feußner, H., Schmauss, B., Hornegger, J.: Time-of-Flight 3-D Endoscopy. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009, Part I. LNCS, vol. 5761, pp. 467–474. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Haase, S., Forman, C., Kilgus, T., Bammer, R., Maier-Hein, L., Hornegger, J.: Tof/rgb sensor fusion for augmented 3d endoscopy using a fully automatic calibration scheme. In: Tolxdorff, T., Deserno, T.M., Handels, H., Meinzer, H.P. (eds.) Bildverarbeitung für die Medizin 2012, pp. 467–474. Springer, Heidelberg (2012)Google Scholar
  6. 6.
    Haase, S., Wasza, J., Kilgus, T., Hornegger, J.: Laparoscopic instrument localization using a 3-d time-of-flight/rgb endoscope. In: WACV 2013, pp. 449–454 (2013)Google Scholar
  7. 7.
    Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process Mag. 20(3), 21–36 (2003)CrossRefGoogle Scholar
  8. 8.
    De Smet, V., Namboodiri, V., Van Gool, L.: Super-resolution techniques for minimally invasive surgery. In: Proceedings AE-CAI 2011, pp. 41–50 (2011)Google Scholar
  9. 9.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: Depth superresolution for tof 3d shape scanning. In: CVPR 2009, pp. 343–350 (2009)Google Scholar
  10. 10.
    Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14(2), 47–57 (2004)CrossRefGoogle Scholar
  11. 11.
    Tipping, M.E., Bishop, C.M.: Bayesian image super-resolution. In: Adv. Neural Inf. Process. Syst., pp. 1303–1310. MIT Press (2003)Google Scholar
  12. 12.
    Fransens, R., Strecha, C., Van Gool, L.: Optical flow based super-resolution: A probabilistic approach. Comput. Vis. Image Underst. 106(1), 106–115 (2007)CrossRefGoogle Scholar
  13. 13.
    Nabney, I.T.: NETLAB: Algorithms for Pattern Recognition, 1st edn. Advances in Pattern Recognition. Springer (2002)Google Scholar
  14. 14.
    Liu, C.: Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. PhD thesis, Massachusetts Institute of Technology (2009)Google Scholar
  15. 15.
    Capel, D.: Image mosaicing and super-resolution. PhD thesis, Oxford (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Thomas Köhler
    • 1
    • 2
  • Sven Haase
    • 1
  • Sebastian Bauer
    • 1
  • Jakob Wasza
    • 1
  • Thomas Kilgus
    • 3
  • Lena Maier-Hein
    • 3
  • Hubertus Feußner
    • 4
  • Joachim Hornegger
    • 1
    • 2
  1. 1.Pattern Recognition LabFriedrich-Alexander-Universität Erlangen-NürnbergGermany
  2. 2.Erlangen Graduate School in Advanced Optical Technologies (SAOT)Germany
  3. 3.Div. Medical and Biological Informatics Junior Group: Computer-Assisted InterventionsGerman Cancer Research Center (DKFZ) HeidelbergGermany
  4. 4.Minimally Invasive Therapy and InterventionTechnical University of MunichGermany

Personalised recommendations