Skip to main content
SpringerLink
Log in

MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13666))

Included in the following conference series:

Abstract

Partial occlusion effects are a phenomenon that blurry objects near a camera are semi-transparent, resulting in partial appearance of occluded background. However, it is challenging for existing bokeh rendering methods to simulate realistic partial occlusion effects due to the missing information of the occluded area in an all-in-focus image. Inspired by the learnable 3D scene representation, Multiplane Image (MPI), we attempt to address the partial occlusion by introducing a novel MPI-based high-resolution bokeh rendering framework, termed MPIB. To this end, we first present an analysis on how to apply the MPI representation to bokeh rendering. Based on this analysis, we propose an MPI representation module combined with a background inpainting module to implement high-resolution scene representation. This representation can then be reused to render various bokeh effects according to the controlling parameters. To train and test our model, we also design a ray-tracing-based bokeh generator for data generation. Extensive experiments on synthesized and real-world images validate the effectiveness and flexibility of this framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadie, G., McAuley, S., Golubev, E., Hill, S., Lagarde, S.: Advances in real-time rendering in games. In: ACM SIGGRAPH 2018 Courses (2018)

    Google Scholar 

  2. Barron, J.T., Adams, A., Shih, Y., Hernández, C.: Fast bilateral-space stereo for synthetic defocus. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4466–4474 (2015)

    Google Scholar 

  3. Busam, B., Hog, M., McDonagh, S., Slabaugh, G.: SteReFo: efficient image refocusing with stereo vision. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (2019)

    Google Scholar 

  4. Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, p. II. IEEE (2003)

    Google Scholar 

  5. Dutta, S., Das, S.D., Shah, N.A., Tiwari, A.K.: Stacked deep multi-scale hierarchical network for fast bokeh effect rendering from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2398–2407 (2021)

    Google Scholar 

  6. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems (NIPS), vol. 27 (2014)

    Google Scholar 

  7. Hach, T., Steurer, J., Amruth, A., Pappenheim, A.: Cinematic bokeh rendering for real scenes. In: Proceedings of the European Conference on Visual Media Production (CVMP), pp. 1–10 (2015)

    Google Scholar 

  8. Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Trans. Graph. (TOG) 26(3), 4-es (2007)

    Google Scholar 

  9. Ignatov, A., Patel, J., Timofte, R.: Rendering natural camera bokeh effect with deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 418–419 (2020)

    Google Scholar 

  10. Ignatov, A., et al.: Aim 2019 challenge on bokeh effect synthesis: methods and results. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 3591–3598. IEEE (2019)

    Google Scholar 

  11. Ignatov, A., et al.: AIM 2020 challenge on rendering realistic bokeh. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 213–228. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_13

    Chapter  Google Scholar 

  12. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (TOG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (ICLR) (2014)

    Google Scholar 

  14. Lee, S., Eisemann, E., Seidel, H.P.: Real-time lens blur effects and focus control. ACM Trans. Graph. (TOG) 29(4), 1–7 (2010)

    Google Scholar 

  15. Levoy, M.: Efficient ray tracing of volume data. ACM Trans. Graph. (TOG) 9(3), 245–261 (1990)

    Article  MATH  Google Scholar 

  16. Li, J., Feng, Z., She, Q., Ding, H., Wang, C., Lee, G.H.: MINE: towards continuous depth MPI with nerf for novel view synthesis. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 12578–12588 (2021)

    Google Scholar 

  17. Lin, S., Ryabtsev, A., Sengupta, S., Curless, B.L., Seitz, S.M., Kemelmacher-Shlizerman, I.: Real-time high-resolution background matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8762–8771 (2021)

    Google Scholar 

  18. Liu, G., Reda, F.A., Shih, K.J., Wang, T.C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 85–100 (2018)

    Google Scholar 

  19. Liu, H., Jiang, B., Song, Y., Huang, W., Yang, C.: Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 725–741. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_43

    Chapter  Google Scholar 

  20. Mildenhall, B., et al.: Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (TOG) 38(4), 1–14 (2019)

    Article  Google Scholar 

  21. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  22. Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: EdgeConnect: structure guided image inpainting using edge prediction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (2019)

    Google Scholar 

  23. Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5437–5446 (2020)

    Google Scholar 

  24. Paszke, A., et al.: Automatic differentiation in PyTorch. In: Advances in Neural Information Processing Systems Workshops (NIPSW) (2017)

    Google Scholar 

  25. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536–2544 (2016)

    Google Scholar 

  26. Peng, J., Luo, X., Xian, K., Cao, Z.: Interactive portrait bokeh rendering system. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 2923–2927. IEEE (2021)

    Google Scholar 

  27. Porter, T., Duff, T.: Compositing digital images. ACM Trans. Graph. (TOG) 253–259 (1984)

    Google Scholar 

  28. Qian, M., et al.: BGGAN: bokeh-glass generative adversarial network for rendering realistic bokeh. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12537, pp. 229–244. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_14

    Chapter  Google Scholar 

  29. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 12179–12188 (2021)

    Google Scholar 

  30. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  31. Schedl, D.C., Wimmer, M.: A layered depth-of-field method for solving partial occlusion. J. WSCG 20(3), 239–246 (2012)

    Google Scholar 

  32. Shen, X., et al.: Automatic portrait segmentation for image stylization. Comput. Graph. Forum 35(2), 93–102 (2016)

    Article  Google Scholar 

  33. Shen, X., Tao, X., Gao, H., Zhou, C., Jia, J.: Deep automatic portrait matting. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 92–107. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_6

    Chapter  Google Scholar 

  34. Shih, M.L., Su, S.Y., Kopf, J., Huang, J.B.: 3D photography using context-aware layered depth inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8028–8038 (2020)

    Google Scholar 

  35. Srinivasan, P.P., Tucker, R., Barron, J.T., Ramamoorthi, R., Ng, R., Snavely, N.: Pushing the boundaries of view extrapolation with multiplane images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 175–184 (2019)

    Google Scholar 

  36. Suvorov, R., et al.: Resolution-robust large mask inpainting with Fourier convolutions. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2149–2159 (2022)

    Google Scholar 

  37. Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 551–560 (2020)

    Google Scholar 

  38. Wadhwa, N., et al.: Synthetic depth-of-field with a single-camera mobile phone. ACM Trans. Graph. (TOG) 37(4), 1–13 (2018)

    Article  Google Scholar 

  39. Wang, L., et al.: DeepLens: shallow depth of field from a single image. ACM Trans. Graph. (TOG) 37(6), 1–11 (2018)

    Google Scholar 

  40. Wu, J., Zheng, C., Hu, X., Xu, F.: Rendering realistic spectral bokeh due to lens stops and aberrations. Vis. Comput. 29(1), 41–52 (2013)

    Article  Google Scholar 

  41. Xian, K., Peng, J., Zhang, C., Lu, H., Cao, Z.: Ranking-based salient object detection and depth prediction for shallow depth-of-field. Sensors 21(5), 1815 (2021)

    Article  Google Scholar 

  42. Xiao, L., Kaplanyan, A., Fix, A., Chapman, M., Lanman, D.: DeepFocus: learned image synthesis for computational displays. ACM Trans. Graph. (TOG) 37(6), 1–13 (2018)

    Article  Google Scholar 

  43. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6721–6729 (2017)

    Google Scholar 

  44. Yang, Y., Lin, H., Yu, Z., Paris, S., Yu, J.: Virtual DSLR: high quality dynamic depth-of-field synthesis on mobile platforms. Electron. Imaging 2016(18), 1–9 (2016)

    Article  Google Scholar 

  45. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4471–4480 (2019)

    Google Scholar 

  46. Yu, X., Wang, R., Yu, J.: Real-time depth of field rendering via dynamic light field generation and filtering. Comput. Graph. Forum 29(7), 2099–2107 (2010)

    Article  Google Scholar 

  47. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018)

    Google Scholar 

  48. Zhang, X., Matzen, K., Nguyen, V., Yao, D., Zhang, Y., Ng, R.: Synthetic defocus and look-ahead autofocus for casual videography. ACM Trans. Graph. (TOG) 38, 1–16 (2019)

    Google Scholar 

  49. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 40(6), 1452–1464 (2017)

    Article  Google Scholar 

  50. Zhou, T., Tucker, R., Flynn, J., Fyffe, G., Snavely, N.: Stereo magnification: learning view synthesis using multiplane images. ACM Trans. Graph. (TOG) 37(4), 1–12 (2018)

    Article  Google Scholar 

  51. Zhou, T., Tulsiani, S., Sun, W., Malik, J., Efros, A.A.: View synthesis by appearance flow. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 286–301. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_18

    Chapter  Google Scholar 

  52. Zhu, M., et al.: Image inpainting by end-to-end cascaded refinement with mask awareness. IEEE Trans. Image Process. (TIP) 30, 4855–4866 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was funded by Adobe.

Author information

Authors and Affiliations

  1. Key Laboratory of Image Processing and Intelligent Control, Ministry of Education, School of AIA, Huazhong University of Science and Technology, Wuhan, China

    Juewen Peng, Xianrui Luo, Hao Lu, Ke Xian & Zhiguo Cao

  2. Adobe Research, San Jose, USA

    Jianming Zhang

Authors
  1. Juewen Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xianrui Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ke Xian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhiguo Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ke Xian .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 18026 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Peng, J., Zhang, J., Luo, X., Lu, H., Xian, K., Cao, Z. (2022). MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_34

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20068-7_34

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20067-0

  • Online ISBN: 978-3-031-20068-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation