Abstract
This paper reviews the second AIM realistic bokeh effect rendering challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world bokeh simulation problem, where the goal was to learn a realistic shallow focus technique using a large-scale EBB! bokeh dataset consisting of 5K shallow/wide depth-of-field image pairs captured using the Canon 7D DSLR camera. The participants had to render bokeh effect based on only one single frame without any additional data from other cameras or sensors. The target metric used in this challenge combined the runtime and the perceptual quality of the solutions measured in the user study. To ensure the efficiency of the submitted models, we measured their runtime on standard desktop CPUs as well as were running the models on smartphone GPUs. The proposed solutions significantly improved the baseline results, defining the state-of-the-art for practical bokeh effect rendering problem.
A. Ignatov and R. Timofte are the challenge organizers, while the other authors participated in the challenge.
The Appendix A contains the authors’ teams and affiliations.
AIM 2020 webpage: https://data.vision.ee.ethz.ch/cvl/aim20/
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Ancuti, C.O., Ancuti, C., Timofte, R., et al.: Ntire 2019 challenge on image dehazing: Methods and results. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)
Badrinarayanan, V., Kendall, A., Cipolla, R.: SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)
Busam, B., Hog, M., McDonagh, S., Slabaugh, G.: SteReFo: efficient image refocusing with stereo vision. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: Semantic image segmentation with deep convolutional nets, ATROUS convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)
delegate, T.L.G. https://www.tensorflow.org/lite/performance/gpu
El Helou, M., et al.: AIM 2020: scene relighting and illumination estimation challenge. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 499–518. Springer, Cham (2020)
Fuoli, D., et al.: AIM 2020 challenge on video extreme super-resolution: Methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12538, pp. 57–81. Springer, Cham (2020)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)
L.B. in the new Google Camera app. https://ai.googleblog.com/2014/04/lens-blur-in-new-google-camera-app.html
Hou, Q., Cheng, M.M., Hu, X., Borji, A., Tu, Z., Torr, P.H.: Deeply supervised salient object detection with short connections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3203–3212 (2017)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Ignatov, A., Kobyshev, N., Timofte, R., Vanhoey, K., Van Gool, L.: DSLR-quality photos on mobile devices with deep convolutional networks. In: the IEEE International Conference on Computer Vision (ICCV) (2017)
Ignatov, A., Patel, J., Timofte, R.: Rendering natural camera bokeh effect with deep learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 418–419 (2020)
Ignatov, A., et al.: AIM 2019 challenge on bokeh effect synthesis: methods and results. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3591–3598. IEEE (2019)
Ignatov, A., et al.: AI Benchmark: running deep neural networks on android smartphones. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 288–314. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_19
Ignatov, A., et al.: AI Benchmark: all about deep learning on smartphones in 2019. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3617–3635. IEEE (2019)
Ignatov, A., Timofte, R., et al.: PIRM challenge on perceptual image enhancement on smartphones: report. In: European Conference on Computer Vision Workshops (2018)
Ignatov, A., Timofte, R., et al.: AIM 2020 challenge on learned image signal processing pipeline. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 152–170. Springer, Cham (2020)
Ignatov, A., Timofte, R., et al.: AIM 2020 challenge on rendering realistic bokeh. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 213–228. Springer, Cham (2020)
Ignatov, A., Van Gool, L., Timofte, R.: Replacing mobile camera ISP with a single deep learning model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 536–537 (2020)
Jia, X., De Brabandere, B., Tuytelaars, T., Gool, L.V.: Dynamic filter networks. In: Advances in Neural Information Processing Systems, pp. 667–675 (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Li, Z., Snavely, N.: Megadepth: learning single-view depth prediction from internet photos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2041–2050 (2018)
Lite, T.: https://www.tensorflow.org/lite
Liu, P., Zhang, H., Zhang, K., Lin, L., Zuo, W.: Multi-level wavelet-CNN for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 773–782 (2018)
Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990–4998 (2017)
Luo, X., Peng, J., Xian, K., Wu, Z., Cao, Z.: Bokeh rendering from defocus estimation. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 245–261. Springer, Cham (2020)
Ntavelis, E., et al.: AIM 2020 challenge on image extreme inpainting. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 716–741. Springer, Cham (2020)
Purohit, K., Suin, M., Kandula, P., Ambasamudram, R.: Depth-guided dense dynamic filtering network for bokeh effect rendering. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3417–3426. IEEE (2019)
Qian, M., Qiao, C., Lin, J., Guo, Z., Li, C., Leng, C., Cheng, J.: Bggan: Bokeh-glass generative adversarial network for rendering realistic bokeh. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 229–244. Springer, Cham (2020)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shen, X., et al.: Automatic portrait segmentation for image stylization. In: Computer Graphics Forum, vol. 35, pp. 93–102. Wiley Online Library (2016)
Son, S., Lee, J., Nah, S., Timofte, R., Lee, K.M., et al.: AIM 2020 challenge on video temporal super-resolution. In: European Conference on Computer Vision Workshops (2020)
Timofte, R., Gu, S., Wu, J., Van Gool, L.: Ntire 2018 challenge on single image super-resolution: methods and results. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018
Wadhwa, N., Garg, R., Jacobs, D.E., Feldman, B.E., Kanazawa, N., Carroll, R., Movshovitz-Attias, Y., Barron, J.T., Pritch, Y., Levoy, M.: Synthetic depth-of-field with a single-camera mobile phone. ACM Trans. Graphics (TOG) 37(4), 64 (2018)
Wei, P., Lu, H., Timofte, R., Lin, L., Zuo, W., et al.: AIM 2020 challenge on real image super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 392–422. Springer, Cham (2020)
Xu, N., Price, B., Cohen, S., Huang, T.: Deep image matting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2970–2979 (2017)
Yu, J., et al.: Wide activation for efficient and accurate image super-resolution. arXiv preprint arXiv:1808.08718 (2018)
Zhang, K., et al.: AIM 2020 challenge on efficient super-resolution: methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, LNCS 12537, pp. 5–40. Springer, Cham (2020)
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., Fu, Y.: Image super-resolution using very deep residual channel attention networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 294–310. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_18
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Acknowledgment
We thank the AIM 2020 sponsors: Huawei, MediaTek, Qualcomm, NVIDIA, Google and Computer Vision Lab/ETH Zürich.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
A Appendix 1: Teams and affiliations
A Appendix 1: Teams and affiliations
AIM 2020 Realistic Bokeh Rendering Challenge Team
Title: AIM 2020 Challenge on Rendering Realistic Bokeh
Members: Andrey Ignatov – andrey@vision.ee.ethz.ch,
Radu Timofte – radu.timofte@vision.ee.ethz.ch
Affiliations: Computer Vision Lab, ETH Zurich, Switzerland
Airia-bokeh
Title: BGNet: Bokeh-Glass Network for Rendering Realistic Bokeh
Members: Ming Qian – 20181223053@nuist.edu.cn,
Congyu Qiao, Jiamin Lin, Zhenyu Guo, Chenghua Li,
Cong Leng, Jian Cheng
Affiliations: Nanjing Artificial Intelligence Chip Research, Institute of Automation
Chinese Academy of Sciences (AiRiA), MAICRO, China
AIA-Smart
Title: Bokeh Rendering from Defocus Estimation [28]
Members: Juewen Peng – im.pengjw@gmail.com,
Xianrui Luo, Ke Xian, Zijin Wu, Zhiguo Cao
Affiliations: Huazhong University of Science and Technology, China
CET_CVLab
Title: Photorealistic Bokeh Effect Rendering with Dilated Wavelet CNN
Members: Densen Puthussery – puthusserydensen@gmail.com,
Jiji C V
Affiliations: College of Engineering Trivandrum, India
CET_SP
Title: Bokeh Effect using VGG based Wavelet CNN
Members: Hrishikesh P S – hrishikeshps@cet.ac.in,
Melvin Kuriakose
Affiliations: College of Engineering Trivandrum, India
Team Horizon
Title: Deep Multi-scale Hierarchical Network for Bokeh Effect Rendering
Members: Saikat Dutta – cs18s016@smail.iitm.ac.in,
Sourya Dipta Das, Nisarg A. Shah
Affiliations: Indian Institute of Technology Madras, India
Jadavpur University, India
Indian Institute of Technology Jodhpur, India
IPCV_IITM
Title: Dense Dynamic Filtering Network for Rendering Synthetic Depth-of-Field Effect
Members: Kuldeep Purohit – kuldeeppurohit3@gmail.com,
Praveen Kandula, Maitreya Suin, A. N. Rajagopalan
Affiliations: Indian Institute of Technology Madras, India
CET21_CV
Title: Synthetic Bokeh Effect with Modified UNet
Members: Saagara M B – saagara@cet.ac.in,
Minnu A L
Affiliations: College of Engineering Trivandrum, India
CET_ECE
Title: Bokeh Effect Rendering with Deep Convolutional Neural Network
Members: Sanjana A R – ar.sanjanaar@gmail.com,
Praseeda S
Affiliations: College of Engineering Trivandrum, India
Xuehuapiaopiao-team
Title: Multi-scale Bokeh Rendering Network
Members: Ge Wu – 1047670389@qq.com,
Xueqin Chen, Tengyao Wang
Affiliations: None
Terminator
Title: Simulating Realistic Bokeh Rendering with an Improved Dataset and Robust Network
Members: Max Zheng – 1843639867@qq.com,
Hulk Wong, Jay Zou
Affiliations: None
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ignatov, A. et al. (2020). AIM 2020 Challenge on Rendering Realistic Bokeh. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12537. Springer, Cham. https://doi.org/10.1007/978-3-030-67070-2_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-67070-2_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67069-6
Online ISBN: 978-3-030-67070-2
eBook Packages: Computer ScienceComputer Science (R0)