Advertisement

Journal of Real-Time Image Processing

, Volume 5, Issue 2, pp 109–120 | Cite as

Rendering techniques for mixed reality

  • Thomas GierlingerEmail author
  • Daniel Danch
  • André Stork
Special Issue

Abstract

In mixed reality (MR) design review, the aesthetics of a virtual prototype is assessed by integrating a virtual model into a real-world environment and inspecting the interaction between the model and the environment (lighting, shadows and reflections) from different points of view. The visualization of the virtual model has to be as realistically as possible to provide a solid basis for this assessment and interactive rendering speed is mandatory to allow the designer to examine the scene from arbitrary positions. In this article we present a real-time rendering engine specifically tailored to the needs of MR visualization. The renderer utilizes pre-computed radiance transfer to calculate dynamic soft-shadows, high dynamic range images and image-based lighting to capture incident real-world lighting, approximate bidirectional texture functions to render materials with self-shadowing, and frame post-processing filters (bloom filter and an adaptive tone mapping operator). The proposed combination of rendering techniques provides a trade-off between rendering quality and required computing resources which enables high quality rendering in mobile MR scenarios. The resulting image fidelity is superior to radiosity-based techniques because glossy materials and dynamic environment lighting with soft-shadows are supported. Ray tracing-based techniques provide higher quality images than the proposed system, but they require a cluster of computers to achieve interactive frame rates which prevents these techniques from being used in mobile MR (especially outdoor) scenarios. The renderer was developed in the European research project IMPROVE (FP6-IST-2-004785) and is currently extended in the MAXIMUS project (FP7-ICT-1-217039) where hybrid rendering techniques which fuse PRT and ray tracing are developed.

Keywords

Mixed reality rendering Real-time rendering High dynamic range imaging Image-based lighting Pre-computed radiance transfer Approximate bidirectional texture function Adaptive tone mapping 

Notes

Acknowledgments

The research leading to these results has received funding from the European Community’s Sixth and Seventh Framework Programme (under grant agreement FP6-IST-2-004785 and FP7-ICT-1-217039).

References

  1. 1.
    ARIS Project: European research project ARIS, augmented reality image synthesis through illumination reconstruction and its integration in interactive and shared mobile ar-systems for e-(motion)-commerce applications. http://aris-ist.intranet.gr (2004)
  2. 2.
    Blinn, J.F.: Simulation of wrinkled surfaces. In: SIGGRAPH ’78: Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques, pp. 286–292. ACM Press, New York (1978). doi:http://doi.acm.org/10.1145/800248.507101
  3. 3.
    Blinn, J.F., Newell, M.E.: Texture and reflection in computer generated images. In: SIGGRAPH ’76: Proceedings of the 3rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 266–266. ACM Press, New York (1976). doi:http://doi.acm.org/10.1145/563274.563322
  4. 4.
    Caustic Graphics.: (2009). http://caustic.com/docs_faq.php
  5. 5.
    Cook, R.L., Porter, T., Carpenter, L.: Distributed ray tracing. In: SIGGRAPH ’84: Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, pp. 137–145. ACM Press, New York (1984). doi:http://doi.acm.org/10.1145/800031.808590
  6. 6.
    Dana, K.J., Nayar, S.K., Ginneken, B.V., Koenderink, J.J.: Reflectance and texture of real-world surfaces. In: CVPR ’97: Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR ’97), p. 151. IEEE Computer Society, Washington, DC (1997)Google Scholar
  7. 7.
    Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: SIGGRAPH ’98: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, pp. 189–198. ACM Press, New York (1998). doi:http://doi.acm.org/10.1145/280814.280864
  8. 8.
    Drettakis, G., Sillion, F.X.: Interactive update of global illumination using a line-space hierarchy. In: SIGGRAPH ’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp. 57–64. ACM Press, New York (1997). doi:http://doi.acm.org/10.1145/258734.258772
  9. 9.
    Fiat Elasis.: (2009). http://www.elasis.it
  10. 10.
    Franke, T., Jung, Y.: Precomputed radiance transfer for x3d based mixed reality applications. In: Web3D ’08: Proceedings of the 13th International Symposium on 3D Web Technology, pp. 7–10. ACM Press, New York (2008). doi:http://doi.acm.org/10.1145/1394209.1394213
  11. 11.
    Goral, C.M., Torrance, K.E., Greenberg, D.P., Battaile, B.: Modeling the interaction of light between diffuse surfaces. In: SIGGRAPH ’84: Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, pp 213–222. ACM Press, New York (1984). doi:http://doi.acm.org/10.1145/800031.808601
  12. 12.
    Hanrahan, P., Salzman, D., Aupperle, L.: A rapid hierarchical radiosity algorithm. In: SIGGRAPH ’91: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, pp. 197–206. ACM Press, New York (1991). doi:http://doi.acm.org/10.1145/122718.122740
  13. 13.
    HDRShop: (2009). http://www.hdrshop.com
  14. 14.
    Jensen, H.W.: Realistic image synthesis using photon mapping. A. K. Peters Ltd, Natick (2001)Google Scholar
  15. 15.
    Kajiya, J.T.: The rendering equation. SIGGRAPH Comput. Graph. 20(4), 143–150 (1986). doi:http://doi.acm.org/10.1145/15886.15902
  16. 16.
    Kautz, J.: Approximate bidirectional texture functions. In: Pharr, M. (ed.) GPU Gems 2, pp. 177–187. Addison-Wesley, Reading (2005)Google Scholar
  17. 17.
    Kautz, J., McCool, M.D.: Approximation of glossy reflection with prefiltered environment maps. In: Graphics Interface, pp. 119–126 (2000)Google Scholar
  18. 18.
    Keller, A.: Instant radiosity. In: SIGGRAPH ’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pp 49–56. ACM Press, New York (1997). doi:http://doi.acm.org/10.1145/258734.258769
  19. 19.
    Klinker, G., Dutoit, A.H., Bauer, M., Bayer, J., Novak, V., Matzke, D.: Fata morgana—a presentation system for product design. In: ISMAR ’02: Proceedings of the 1st International Symposium on Mixed and Augmented Reality, p. 76. IEEE Computer Society, Washington, DC (2002)Google Scholar
  20. 20.
    Lehtinen, J., Kautz, J.: Matrix radiance transfer. In: I3D ’03: Proceedings of the 2003 Symposium on Interactive 3D Graphics, pp. 59–64. ACM Press, New York (2003). doi:http://doi.acm.org/10.1145/641480.641495
  21. 21.
    Müller, G., Meseth, J., Klein, R.: Fast environmental lighting for local-pca encoded btfs. In: CGI ’04: Proceedings of the Computer Graphics International, pp. 198–205. IEEE Computer Society, Washington, DC (2004). doi:http://dx.doi.org/10.1109/CGI.2004.49
  22. 22.
    Nicodemus, F.E., Richmond, J.C., Hsia, J.J., Ginsberg, I.W., Limperis, T.: Geometrical considerations and nomenclature for reflectance. Final Report National Bureau of Standards, Washington, DC (1977)Google Scholar
  23. 23.
    O’Malley, S.M.: A simple, effective system for automated capture of high dynamic range images. In: ICVS ’06: Proceedings of the Fourth IEEE International Conference on Computer Vision Systems, p. 15. IEEE Computer Society, Washington, DC (2006). doi:http://dx.doi.org/10.1109/ICVS.2006.8
  24. 24.
    OpenGL Architecture Review Board, Shreiner, D., Woo, M., Neider, J., Davis, T.: OpenGL(R) Programming Guide: The Official Guide to Learning OpenGL(R), Version 2.1. Addison-Wesley, Reading (2007)Google Scholar
  25. 25.
    OpenSG.: (2009). http://opensg.vrsource.org
  26. 26.
    Page\Park Architects: (2009). http://www.pagepark.co.uk
  27. 27.
  28. 28.
    Reinhard, E., Stark, M., Shirley, P., Ferwerda, J.: Photographic tone reproduction for digital images. In: SIGGRAPH ’02: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 267–276. ACM Press, New York (2002). doi:http://doi.acm.org/10.1145/566570.566575
  29. 29.
    Rost, R.J.: OpenGL(R) Shading Language. Addison-Wesley, Redwood City (2004)Google Scholar
  30. 30.
    Shirley, P., Morley, R.K.: Realistic Ray Tracing. A. K. Peters Ltd, Natick (2003)Google Scholar
  31. 31.
    Sillion, F.X., Arvo, J.R., Westin, S.H., Greenberg, D.P.: A global illumination solution for general reflectance distributions. In: SIGGRAPH ’91: Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–196. ACM Press, New York (1991). doi:http://doi.acm.org/10.1145/122718.122739
  32. 32.
    Sloan, P.P., Kautz, J., Snyder, J.: Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In: SIGGRAPH ’02: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 527–536. ACM Press, New York (2002). doi:http://doi.acm.org/10.1145/566570.566612
  33. 33.
    Spheron: (2009). http://www.spheron.com
  34. 34.
    Wald, I., Dietrich, A., Benthin, C., Efremov, A., Dahmen, T., Günther, J., Havran, V., Seidel, H.P., Slusallek, P.: Applying ray tracing for virtual reality and industrial design. In: Proceedings of the 2006 IEEE Symposium on Interactive Ray Tracing, pp. 177–185 (2006)Google Scholar
  35. 35.
    Wallner, G.: GPU radiosity for triangular meshes with support of normal mapping and arbitrary light distributions. J. WSCG 16(1–3), 1–8 (2008)Google Scholar
  36. 36.
    Whitted, T.: An improved illumination model for shaded display. Commun. ACM 23(6), 343–349 (1980). doi:http://doi.acm.org/10.1145/358876.358882 Google Scholar
  37. 37.
    Williams, L.: Casting curved shadows on curved surfaces. SIGGRAPH Comput. Graph. 12(3), 270–274 (1978). doi:http://doi.acm.org/10.1145/965139.807402
  38. 38.
    Zhou, K., Hou, Q., Wang, R., Guo, B.: Real-time kd-tree construction on graphics hardware. In: SIGGRAPH Asia’08: ACM SIGGRAPH Asia 2008 papers, pp. 1–11. ACM Press, New York (2008). doi:http://doi.acm.org/10.1145/1457515.1409079

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  1. 1.Fraunhofer IGDDarmstadtGermany
  2. 2.TU DarmstadtDarmstadtGermany

Personalised recommendations