The Visual Computer

, Volume 30, Issue 1, pp 45–58 | Cite as

Racking focus and tracking focus on live video streams: a stereo solution

  • Zhan Yu
  • Xuan Yu
  • Christopher Thorpe
  • Scott Grauer-Gray
  • Feng Li
  • Jingyi Yu
Original Article

Abstract

The ability to produce dynamic Depth of Field effects in live video streams was until recently a quality unique to movie cameras. In this paper, we present a computational camera solution coupled with real-time GPU processing to produce runtime dynamic Depth of Field effects. We first construct a hybrid-resolution stereo camera with a high-res/low-res camera pair. We recover a low-res disparity map of the scene using GPU-based Belief Propagation, and subsequently upsample it via fast Cross/Joint Bilateral Upsampling. With the recovered high-resolution disparity map, we warp the high-resolution video stream to nearby viewpoints to synthesize a light field toward the scene. We exploit parallel processing and atomic operations on the GPU to resolve visibility when multiple pixels warp to the same image location. Finally, we generate racking focus and tracking focus effects from the synthesized light field rendering. All processing stages are mapped onto NVIDIA’s CUDA architecture. Our system can produce racking and tracking focus effects for the resolution of 640×480 at 15 fps.

Keywords

Dynamic depth of field Racking focus Tracking focus Belief propagation Cross bilateral Filtering Light field CUDA 

References

  1. 1.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001) CrossRefGoogle Scholar
  2. 2.
    Brunton, A., Shu, C., Roth, G.: Belief propagation on the GPU for stereo vision. In: The 3rd Canadian Conference on Computer and Robot Vision (2006) Google Scholar
  3. 3.
    Felzenszwalb, P., Huttenlocher, D.: Efficient belief propagation for early vision. In: CVPR (2004) Google Scholar
  4. 4.
    Grauer-Gray, S., Kambhamettu, C., Palaniappan, K.: GPU implementation of belief propagation using CUDA for cloud tracking and reconstruction. In: 2008 IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS 2008), pp. 1–4 (2008) CrossRefGoogle Scholar
  5. 5.
    Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Anal. Mach. Intell. 26(2), 147–159 (2004) CrossRefGoogle Scholar
  6. 6.
    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. In: SIGGRAPH (2007) Google Scholar
  7. 7.
    Lee, S., Eisemann, E., Seidel, H.P.: Depth-of-field rendering with multiview synthesis. In: SIGGRAPH Asia (2009) Google Scholar
  8. 8.
    Lee, S., Kim, G.J., Choi, S.: Real-time depth-of-field rendering using anisotropically filtered mipmap interpolation. IEEE Trans. Vis. Comput. Graph. 15(3), 453–464 (2009) CrossRefGoogle Scholar
  9. 9.
    Levin, A., Hasinoff, S.W., Green, P., Durand, F., Freeman, W.T.: 4d frequency analysis of computational cameras for depth of field extension. ACM Trans. Graph. 28 (2009) Google Scholar
  10. 10.
    Li, F., Yu, J., Chai, J.: A hybrid camera for motion deblurring and depth map super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2008) Google Scholar
  11. 11.
    Mcmillan, L., Yang, J.C., Yang, J.C.: A light field camera for image based rendering (2000) Google Scholar
  12. 12.
    Ng, R.: Fourier slice photography. In: SIGGRAPH (2005) Google Scholar
  13. 13.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Stanford tech report ctsr 2005-02 light field photography with a hand-held plenoptic camera Google Scholar
  14. 14.
    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. Int. J. Comput. Vis. 81(1), 24–52 (2009) CrossRefGoogle Scholar
  15. 15.
    Sawhney, H.S., Guo, Y., Hanna, K., Kumar, R.: Hybrid stereo camera: an IBR approach for synthesis of very high resolution stereoscopic image sequences. In: SIGGRAPH, pp. 451–460 (2001) Google Scholar
  16. 16.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002) CrossRefMATHGoogle Scholar
  17. 17.
    Stroebel, L., Compton, J., Current, I., Zakia, R.: Photographic Materials and Processes (1986) Google Scholar
  18. 18.
    Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7), 787–800 (2003) CrossRefGoogle Scholar
  19. 19.
    Vaish, V., Levoy, M., Szeliski, R., Zitnick, C., Kang, S.B.: Reconstructing occluded surfaces using synthetic apertures: stereo, focus and robust measures. In: CVPR (2006) Google Scholar
  20. 20.
    Vaish, V., Wilburn, B., Joshi, N., Levoy, M.: Using plane + parallax for calibrating dense camera arrays. In: CVPR (2004) Google Scholar
  21. 21.
    Wang, H., Sun, M., Yang, R.: Space-time light field rendering. IEEE Trans. Vis. Comput. Graph. 13, 697–710 (2007) CrossRefGoogle Scholar
  22. 22.
    Wang, H., Yang, R.: Towards space: time light field rendering. In: I3D (2005) Google Scholar
  23. 23.
    Wilburn, B., Joshi, N., Vaish, V., Levoy, M., Horowitz, M.: High-speed videography using a dense camera array. In: CVPR (2004) Google Scholar
  24. 24.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graph. 24, 765–776 (2005) CrossRefGoogle Scholar
  25. 25.
    Yang, J.C., Everett, M., Buehler, C., McMillan, L.: A real-time distributed light field camera. In: Proceedings of the 13th Eurographics Workshop on Rendering, EGRW ’02, pp. 77–86 (2002) Google Scholar
  26. 26.
    Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR (2007) Google Scholar
  27. 27.
    Yu, X., Wang, R., Yu, J.: Real-time depth of field rendering via dynamic light field generation and filtering. Comput. Graph. Forum 29(7), 2099–2107 (2010) CrossRefGoogle Scholar
  28. 28.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000) CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Zhan Yu
    • 1
  • Xuan Yu
    • 1
  • Christopher Thorpe
    • 1
  • Scott Grauer-Gray
    • 1
  • Feng Li
    • 1
  • Jingyi Yu
    • 1
  1. 1.University of DelawareNewarkUSA

Personalised recommendations