A GPU-Based Real-Time Algorithm for Virtual Viewpoint Rendering from Multi-video

Chapter
First Online:

Abstract

In this chapter, we propose a novel GPU-based algorithm capable of generating free viewpoints from a network of fixed HD video cameras. This free viewpoint TV system consists of two main subsystems: a real-time depth estimation subsystem, which extracts a disparity map from a network of cameras, and a synthetic viewpoint generation subsystem that uses the disparity map to interpolate new views between the cameras. In this system, we use a space-sweep algorithm to estimate depth information, which is amiable to parallel implementation. The viewpoint generation subsystem generates new synthetic images from 3D vertices and renders them from an arbitrary viewpoint specified by the user. Both steps are computationally extensive, but the computations can be easily divided from each other and thus can be efficiently implemented in parallel using CUDA. The framework is tested using publicly available image sequences published by Microsoft. Experimental results are presented.

Keywords

Real-time free viewpoint television GPU-accelerated algorithms CUDA 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Tanimoto, M.: Overview of free viewpoint television. Signal Processing: Image Communication 21(6), 454–461 (2006), Special issue on multi-view image processing and its application in image-based renderingGoogle Scholar
  2. 2.
    Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 369–374. SIGGRAPH ’00. ACM Press/Addison-Wesley Publishing Co, New York, NY. (2000)Google Scholar
  3. 3.
    Franco, J.S., Boyer, E.: Exact polyhedral visual hulls. In: British Machine Vision Conference (BMVC’03), vol 1, pp. 329–338. Norwich, Royaume-Uni (2003)Google Scholar
  4. 4.
    Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE Comput. Graph. Appl. 27(3), 21–31 (2007)Google Scholar
  5. 5.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010)Google Scholar
  6. 6.
    Starck, J., Hilton, A.: Virtual view synthesis of people from multiple view video sequences. Graph. Models 67(6), 600–620 (2005)CrossRefMATHGoogle Scholar
  7. 7.
    Orman N., Kim H., Sakamoto R., Toriyama T., Kogure K., Lindeman R.: GPU-based optimization of a free-viewpoint video system. In: Proceedings of the 2008 symposium on Interactive 3D graphics and games (2008)Google Scholar
  8. 8.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 43–54. SIGGRAPH ’96. ACM, New York, NY (1996)Google Scholar
  9. 9.
    Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 31–42. SIGGRAPH ’96. ACM, New York, NY (1996)Google Scholar
  10. 10.
    Fujii, T., Kimoto, T., Tanimoto, M.: Ray-space coding for 3d visual communication. In: Picture Coding Symposium (PCS), vol. 2, pp. 447–451 (1996)Google Scholar
  11. 11.
    Mori, Y., Fukushima, N., Yendo, T., Fujii, T., Tanimoto, M.: View generation with 3d warping using depth information for ftv. Image Commun. 24(1–2), 65–72 (2009)Google Scholar
  12. 12.
    Zitnick, C.L., Kang, S.B., Uyttendaele, M., Winder, S., Szeliski, R.: High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23(3), 600–608 (2004)CrossRefGoogle Scholar
  13. 13.
    Yang, R., Welch, G., Bishop, G.: Real-time consensus-based scene reconstruction using commodity graphics hardware. In: Pacific Conference on Computer Graphics and Applications. 225–235 (2002)Google Scholar
  14. 14.
    Suzuki, K., Fukushima, N., Yendo, T., Tehrani, M., Fujii, T., Tanimoto, M.: Parallel processing method for realtime ftv. In: Picture Coding Symposium (PCS), pp. 330–333 (2010)Google Scholar
  15. 15.
    Nozick V., Saito H.: On-line free-viewpoint video: From single to multiple view rendering. Int. J. Autom. Comput. 5, 257–267 (2008)Google Scholar
  16. 16.
    Starck J., Kilner J., Hilton A.: A free-viewpoint renderer. J. Graph. GPU Game Tools 14 57–72 (2009)Google Scholar
  17. 17.
    Collins, R.: A space-sweep approach to true multi-image matching. In: Computer Vision and Pattern Recognition. IEEE Computer Society Conference on proceedings CVPR ’96, pp. 358–363 (1996)Google Scholar
  18. 18.
    NVIDIA Corporation: NVIDIA CUDA C programming guide (2012) Version 5.0.Google Scholar
  19. 19.
    Stam, J.: Stereo imaging with cuda. (2008)Google Scholar
  20. 20.
    Zhou, X., Boulanger, P.: Radiometric invariant stereo matching based on relative gradients. In: 19th IEEE International Conference on Image Processing (ICIP), pp. 2989–2992 (2012)Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2015

Authors and Affiliations

  1. 1.Computing Science DepartmentUniversity of AlbertaEdmontonCanada

Personalised recommendations