The Visual Computer

, Volume 31, Issue 6–8, pp 1023–1032 | Cite as

Real-time adaptive content retargeting for live multi-view capture and light field display

  • Vamsi Kiran Adhikarla
  • Fabio Marton
  • Tibor Balogh
  • Enrico Gobbetti
Original Article

Abstract

The discrete nature of multiprojector light field displays results in aliasing when rendering scene points at depths outside the supported depth of field causing visual discomfort. We propose an efficient on-the-fly content-aware real-time depth retargeting algorithm for live 3D light field video to increase the quality of visual perception on a cluster-driven multiprojector light field display. The proposed algorithm is embedded in an end-to-end real-time system capable of capturing and reconstructing light field from multiple calibrated cameras on a full horizontal parallax light field display. By automatically detecting salient regions of a scene, we solve an optimization to derive a non-linear operator to fit the whole scene within the comfortable viewing range of the light field display. We evaluate the effectiveness of our approach on synthetic and real world scenes.

Keywords

On-the-fly depth retargeting GPU  Multiprojector light field display  Visually enhanced live 3D video  Multi-view capture and display 

Notes

Acknowledgments

This research has received funding from the DIVA Marie Curie Action of the People programme of the EU FP7/2007- 2013/ Program under REA Grant agreement 290227. The support of the TAMOP-4.2.1.B-11/2/KMR-2011-0002 is kindly acknowledged.

Supplementary material

Supplementary material 1 (mp4 21330 KB)

References

  1. 1.
    Agus, M., Gobbetti, E., Iglesias Guitián, J.A., Marton, F., Pintore, G.: GPU accelerated direct volume rendering on an interactive light field display. Comput. Graph. Forum 27(2), 231–240 (2008)CrossRefGoogle Scholar
  2. 2.
    Agus, M., Gobbetti, E., Jaspe Villanueva, A., Pintore, G., Pintus, R.: Automatic geometric calibration of projector-based light-field displays. In: Proc. Eurovis Short Papers, pp. 1–5 (2013)Google Scholar
  3. 3.
    Balogh, T., Kovács, P.: Real-time 3D light field transmission. In: SPIE, vol. 7724, p. 5 (2010)Google Scholar
  4. 4.
    Birklbauer, C., Bimber, O.: Light-field retargeting. Comput. Graph. Forum 31(2pt1), 295–303 (2012)CrossRefGoogle Scholar
  5. 5.
    Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: Proceedings of SIGGRAPH, pp. 307–318 (2000)Google Scholar
  6. 6.
    Didyk, P., Ritschel, T., Eisemann, E., Myszkowski, K., Seidel, H.P.: A perceptual model for disparity. ACM Trans. Graph. 30(4), 96:1–96:10 (2011)Google Scholar
  7. 7.
    Graf, D., Panozzo, D., Sorkine-Hornung, O.: Mobile image retargeting. In: Proceedings of VMV (2013)Google Scholar
  8. 8.
    Iglesias Guitián, J., Gobbetti, E., Marton, F.: View-dependent exploration of massive volumetric models on large scale light field displays. Vis. Comput. 26(6–8), 1037–1047 (2010)CrossRefGoogle Scholar
  9. 9.
    Jones, A., McDowall, I., Yamada, H., Bolas, M.T., Debevec, P.E.: Rendering for an interactive 360 degree light field display. ACM Trans. Graph 26(3), 40 (2007)CrossRefGoogle Scholar
  10. 10.
    Kim, C., Hornung, A., Heinzle, S., Matusik, W., Gross, M.: Multi-perspective stereoscopy from light fields. ACM Trans. Graph. 30(6), 190:1–190:10 (2011)Google Scholar
  11. 11.
    Kraevoy, V., Sheffer, A., Shamir, A., Cohen-Or, D.: Non-homogeneous resizing of complex models. ACM Trans. Graph. 27(5), 1–9 (2008)Google Scholar
  12. 12.
    Kubota, A., Smolic, A., Magnor, M., Tanimoto, M., Chen, T., Zhang, C.: Multiview imaging and 3DTV. IEEE Signal Process. Mag. 24(6), 10 (2007)CrossRefGoogle Scholar
  13. 13.
    Kunita, Y., Ueno, M., Tanaka, K.: Layered probability maps: basic framework and prototype system. In: Proceedings of VRST, pp. 181–188 (2006)Google Scholar
  14. 14.
    Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.: Nonlinear disparity mapping for stereoscopic 3d. ACM Trans. Graph. 29(4), 75:1–75:10 (2010)CrossRefGoogle Scholar
  15. 15.
    Marton, F., Gobbetti, E., Bettio, F., Iglesias Guitián, J., Pintus, R.: A real-time coarse-to-fine multiview capture system for all-in-focus rendering on a light-field display. In: Proceedings of 3DTV-CON, pp. 1–4 (2011)Google Scholar
  16. 16.
    Masia, B., Wetzstein, G., Aliaga, C., Raskar, R., Gutierrez, D.: Display adaptive 3d content remapping. Comput. Graph. 37(8), 983–996 (2013)CrossRefGoogle Scholar
  17. 17.
    Masia, B., Wetzstein, G., Didyk, P., Gutierrez, D.: A survey on computational displays: pushing the boundaries of optics, computation, and perception. Comput. Graph. 37(8), 1012–1038 (2013)CrossRefGoogle Scholar
  18. 18.
    Matusik, W., Pfister, H.: 3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23(3), 814–824 (2004)CrossRefGoogle Scholar
  19. 19.
    Smolic, A.: 3D video and free viewpoint video-from capture to display. Pattern Recogn. 44(9), 1958–1968 (2010)Google Scholar
  20. 20.
    Taguchi, Y., Koike, T., Takahashi, K., Naemura, T.: TransCAIP: a live 3D TV system using a camera array and an integral photography display with interactive control of viewing parameters. IEEE Trans. Vis. Comput. Graph. 15, 841–852 (2009)CrossRefGoogle Scholar
  21. 21.
    Taguchi, Y., Takahashi, K., Naemura, T.: Real-time all-in-focus video-based rendering using a network camera array. In: Proceedings of 3DPVT, pp. 241–244 (2008)Google Scholar
  22. 22.
    Yang, R., Huang, X., Li, S., Jaynes, C.: Toward the light field display: autostereoscopic rendering via a cluster of projectors. IEEE Trans. Vis. Comput. Graph. 14, 84–96 (2008)CrossRefGoogle Scholar
  23. 23.
    Yang, R., Welch, G., Bishop, G.: Real-time consensus-based scene reconstruction using commodity graphics hardware. In: Proceedings of Pacific Conference on Computer Graphics and Applications, pp. 225–234 (2002)Google Scholar
  24. 24.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Proceedings of ECCV, pp. 151–158 (1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Vamsi Kiran Adhikarla
    • 1
    • 3
  • Fabio Marton
    • 2
  • Tibor Balogh
    • 3
  • Enrico Gobbetti
    • 2
  1. 1.Pázmány Péter Catholic UniversityBudapestHungary
  2. 2.CRS4PulaItaly
  3. 3.HolografikaBudapestHungary

Personalised recommendations