Abstract
Volume visualization pipelines have the potential to be improved by the use of light field display technology, allowing enhanced perceptual qualities. However, these displays will require a significant increase in pixels to be rendered at interactive rates. Volume rendering makes use of ray-tracing techniques, which makes this resolution increase challenging for modest hardware. We demonstrate in this work an approach to synthesize the majority of the viewpoints in the light field using a small set of rendered viewpoints via a convolutional neural network. We show that synthesis performance can be further improved by allowing the network access to the volume data itself. To perform this efficiently, we propose a range of approaches and evaluate them against two datasets collected for this task. These approaches all improve synthesis performance and avoid the use of expensive 3D convolutional operations. With this approach, we improve light field volume rendering times by a factor of 8 for our test case.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agus, M., et al.: An interactive 3D medical visualization system based on a light field display. Vis. Comput. 25(9), 883–893 (2009). https://doi.org/10.1007/s00371-009-0311-y
Agus, M., Gobbetti, E., Guitián, J.A.I., Marton, F., Pintore, G.: GPU accelerated direct volume rendering on an interactive light field display. Comput. Graph. Forum 27(2), 231–240 (2008). https://doi.org/10.1111/j.1467-8659.2008.01120.x
Bilen, H., Fernando, B., Gavves, E., Vedaldi, A., Gould, S.: Dynamic image networks for action recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3034–3042 (2016). https://doi.org/10.1109/CVPR.2016.331
Birklbauer, C., Bimber, O.: Light-field supported fast volume rendering. In: ACM SIGGRAPH 2012 Posters on - SIGGRAPH 2012, p. 1. ACM Press, Los Angeles, California (2012). https://doi.org/10.1145/2342896.2343040
Bruton, S., Ganter, D., Manzke, M.: Synthesising light field volumetric visualizations in real-time using a compressed volume representation. In: Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP. pp. 96–105. SciTePress (2019). https://doi.org/10.5220/0007407200960105
Chang, A.X., et al.: ShapeNet: an information-rich 3D model repository. arXiv:1512.03012 [cs] (2015)
Drebin, R.A., Carpenter, L., Hanrahan, P.: Volume rendering. In: Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques, pp. 65–74. SIGGRAPH 1988, ACM, New York, NY, USA (1988). https://doi.org/10.1145/54852.378484
Engelmann, F., Kontogianni, T., Hermans, A., Leibe, B.: Exploring spatial context for 3D semantic segmentation of point clouds. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW). pp. 716–724 (2017). https://doi.org/10.1109/ICCVW.2017.90
Favalora, G.E.: Volumetric 3D displays and application infrastructure. Computer 38(8), 37–44 (2005). https://doi.org/10.1109/MC.2005.276
Fernando, B., Gavves, E.M., Oramas, J., Ghodrati, A., Tuytelaars, T.: Rank pooling for action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 773–787 (2017). https://doi.org/10.1109/TPAMI.2016.2558148
Fishman, E.K., Ney, D.R., Heath, D.G., Corl, F.M., Horton, K.M., Johnson, P.T.: Volume rendering versus maximum intensity projection in CT angiography: what works best, when, and why. Radiographics: A Review Publication of the Radiological Society of North America, Inc 26(3), 905–922 (2006). https://doi.org/10.1148/rg.263055186
Hadwiger, M., Kratz, A., Sigg, C., Bühler, K.: GPU-accelerated deep shadow maps for direct volume rendering. In: Proceedings of the 21st ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, pp. 49–52. GH 2006, ACM, New York, NY, USA (2006). https://doi.org/10.1145/1283900.1283908
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. 35(6), 193:1–193:10 (2016). https://doi.org/10.1145/2980179.2980251
Kühnapfel, U., Çakmak, H.K., Maaß, H.: Endoscopic surgery training using virtual reality and deformable tissue simulation. Comput. Graph. 24(5), 671–682 (2000). https://doi.org/10.1016/S0097-8493(00)00070-4
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 [cs] (2014)
Klokov, R., Lempitsky, V.: Escape from cells: deep kd-networks for the recognition of 3D point cloud models. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 863–872 (2017). https://doi.org/10.1109/ICCV.2017.99
Kniss, J., Kindlmann, G., Hansen, C.: Multidimensional transfer functions for interactive volume rendering. IEEE Trans. Vis. Comput. Graph. 8(3), 270–285 (2002). https://doi.org/10.1109/TVCG.2002.1021579
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., New York (2012)
Lacroute, P., Levoy, M.: Fast volume rendering using a shear-warp factorization of the viewing transformation. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, pp. 451–458. SIGGRAPH 1994, ACM, New York, NY, USA (1994). https://doi.org/10.1145/192161.192283
Lanman, D., Luebke, D.: Near-eye light field displays. In: ACM SIGGRAPH 2013 Emerging Technologies, p. 11:1. SIGGRAPH 2013, ACM, New York, NY, USA (2013). https://doi.org/10.1145/2503368.2503379
Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31–42. SIGGRAPH 1996, ACM, New York, NY, USA (1996). https://doi.org/10.1145/237170.237199
Li, Y., Pirk, S., Su, H., Qi, C.R., Guibas, L.J.: FPNN: field probing neural networks for 3D data. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 307–315. Curran Associates, Inc., New York (2016)
Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 [cs] (2013)
Liu, T.Y.: Learning to rank for information retrieval. Found. Trends Inf. Retr. 3(3), 225–331 (2009). https://doi.org/10.1561/1500000016
Liu, Z., Yeh, R.A., Tang, X., Liu, Y., Agarwala, A.: Video frame synthesis using deep voxel flow. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4473–4481 (2017). https://doi.org/10.1109/ICCV.2017.478
Ljung, P., Krüger, J., Groller, E., Hadwiger, M., Hansen, C.D., Ynnerman, A.: State of the art in transfer functions for direct volume rendering. Comput. Graph. Forum 35(3), 669–691 (2016). https://doi.org/10.1111/cgf.12934
Mora, B., Maciejewski, R., Chen, M., Ebert, D.S.: Visualization and computer graphics on isotropically emissive volumetric displays. IEEE Trans. Vis. Comput. Graph. 15(2), 221–234 (2009). https://doi.org/10.1109/TVCG.2008.99
Mueller, K., Yagel, R.: Fast perspective volume rendering with splatting by utilizing a ray-driven approach. In: Proceedings of Seventh Annual IEEE Visualization 1996, pp. 65–72 (1996). https://doi.org/10.1109/VISUAL.1996.567608
Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive separable convolution. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 261–270 (2017). https://doi.org/10.1109/ICCV.2017.37
Park, E., Yang, J., Yumer, E., Ceylan, D., Berg, A.C.: Transformation-grounded image generation network for novel 3D view synthesis. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 702–711 (2017). https://doi.org/10.1109/CVPR.2017.82
Philips, S., Hlawitschka, M., Scheuermann, G.: Slice-based visualization of brain fiber bundles - a lic-based approach. In: Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, pp. 281–288. SciTePress (2018). https://doi.org/10.5220/0006619402810288
Qi, C.R., Su, H., Kaichun, M., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77–85 (2017). https://doi.org/10.1109/CVPR.2017.16
Qi, C.R., Su, H., Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view CNNs for object classification on 3D data. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5648–5656 (2016). https://doi.org/10.1109/CVPR.2016.609
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5099–5108. Curran Associates, Inc., New York (2017)
Riegler, G., Ulusoy, A.O., Geiger, A.: OctNet: Learning deep 3D representations at high resolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6620–6629 (2017). https://doi.org/10.1109/CVPR.2017.701
Salama, C.R.: GPU-based monte-carlo volume raycasting. In: 15th Pacific Conference on Computer Graphics and Applications (PG 2007). pp. 411–414 (2007). https://doi.org/10.1109/PG.2007.27
Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199–222 (2004). https://doi.org/10.1023/B:STCO.0000035301.49549.88
Srinivasan, P.P., Wang, T., Sreelal, A., Ramamoorthi, R., Ng, R.: Learning to synthesize a 4D RGBD light field from a single image. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2262–2270 (2017). https://doi.org/10.1109/ICCV.2017.246
Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), pp. 945–953. ICCV 2015, IEEE Computer Society, Washington, DC, USA (2015). https://doi.org/10.1109/ICCV.2015.114
Sunden, E., et al.: Inviwo - an extensible, multi-purpose visualization framework. In: 2015 IEEE Scientific Visualization Conference (SciVis), pp. 163–164 (2015). https://doi.org/10.1109/SciVis.2015.7429514
Tuzel, O., Liu, M.-Y., Taguchi, Y., Raghunathan, A.: Learning to rank 3D features. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 520–535. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_34
Wang, P., Li, W., Gao, Z., Zhang, Y., Tang, C., Ogunbona, P.: Scene flow to action map: a new representation for RGB-D based action recognition with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 416–425 (2017). https://doi.org/10.1109/CVPR.2017.52
Wang, P.S., Liu, Y., Guo, Y.X., Sun, C.Y., Tong, X.: O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073608
Wetzstein, G., Lanman, D., Hirsch, M., Raskar, R.: Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Trans. Graph. 31(4), 80:1–80:11 (2012). https://doi.org/10.1145/2185520.2185576
Xie, J., Dai, G., Zhu, F., Wong, E.K., Fang, Y.: Deepshape: deep-learned shape descriptor for 3D shape retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 39(7), 1335–1345 (2017). https://doi.org/10.1109/TPAMI.2016.2596722
Zhang, Y., Dong, Z., Ma, K.: Real-time volume rendering in dynamic lighting environments using precomputed photon mapping. IEEE Trans. Vis. Comput. Graph. 19(8), 1317–1330 (2013). https://doi.org/10.1109/TVCG.2013.17
Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. 37(4), 65:1–65:12 (2018). https://doi.org/10.1145/3197517.3201323
Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18
Acknowledgements
The authors would also like to thank the anonymous referees for their valuable comments and helpful suggestions. This research has been conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/IA/1895.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Bruton, S., Ganter, D., Manzke, M. (2020). Fast Approximate Light Field Volume Rendering: Using Volume Data to Improve Light Field Synthesis via Convolutional Neural Networks. In: Cláudio, A., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2019. Communications in Computer and Information Science, vol 1182. Springer, Cham. https://doi.org/10.1007/978-3-030-41590-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-41590-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41589-1
Online ISBN: 978-3-030-41590-7
eBook Packages: Computer ScienceComputer Science (R0)