Advertisement

AIM 2020 Challenge on Image Extreme Inpainting

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12537)

Abstract

This paper reviews the AIM 2020 challenge on extreme image inpainting. This report focuses on proposed solutions and results for two different tracks on extreme image inpainting: classical image inpainting and semantically guided image inpainting. The goal of track 1 is to inpaint large part of the image with no supervision. Similarly, the goal of track 2 is to inpaint the image by having access to the entire semantic segmentation map of the input. The challenge had 88 and 74 participants, respectively. 11 and 6 teams competed in the final phase of the challenge, respectively. This report gauges current solutions and set a benchmark for future extreme image inpainting methods.

Keywords

Extreme image inpainting Image synthesis Generative modeling 

Notes

Acknowledgements

We thank the AIM 2020 sponsors: Huawei, MediaTek, Qualcomm AI Research, NVIDIA, Google and Computer Vision Lab/ETH Zürich.

References

  1. 1.
    A tensorflow implementation of SRGAN. https://github.com/tensorlayer/srgan.git
  2. 2.
    Alom, M.Z., Hasan, M., Yakopcic, C., Taha, T.M., Asari, V.K.: Recurrent residual convolutional neural network based on U-Net (R2U-Net) for medical image segmentation. arXiv preprint arXiv:1802.06955 (2018)
  3. 3.
    Bai, M., Li, S., Fan, J., Zhou, C., Zuo, L., Na, J., Jeong, M.: Fast light-weight network for extreme image inpainting challenge. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp.742–757 (2020) Google Scholar
  4. 4.
    Bau, D., et al.: Semantic photo manipulation with a generative image prior. ACM Trans. Graph. (Proc. ACM SIGGRAPH) 38(4), 1–11 (2019)Google Scholar
  5. 5.
    Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: thing and stuff classes in context. In: CVPR (2018). https://arxiv.org/abs/1612.03716
  6. 6.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  7. 7.
    El Helou, M., Zhou, R., Süsstrunk, S., Timofte, R., et al.: AIM 2020: scene relighting and illumination estimation challenge. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 499–518 (2020) Google Scholar
  8. 8.
    Fritsche, M., Gu, S., Timofte, R.: Frequency separation for real-world super-resolution. In: 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3599–3608. IEEE (2019)Google Scholar
  9. 9.
    Fuoli, D., Huang, Z., Gu, S., Timofte, R., et al.: AIM 2020 challenge on video extreme super-resolution: methods and results. In: European Conference on Computer Vision Workshops (2020)Google Scholar
  10. 10.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  11. 11.
    Hong, S., Yan, X., Huang, T.E., Lee, H.: Learning hierarchical semantic image manipulation through structured representations. In: Advances in Neural Information Processing Systems, pp. 2713–2723 (2018)Google Scholar
  12. 12.
    Hui, Z., Li, J., Wang, X., Gao, X.: Image fine-grained inpainting. arXiv preprint arXiv:2002.02609 (2020)
  13. 13.
    Ignatov, A., Timofte, R., et al.: AIM 2020 challenge on learned image signal processing pipeline. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 152–170 (2020) Google Scholar
  14. 14.
    Ignatov, A., Timofte, R., et al.: AIM 2020 challenge on rendering realistic bokeh. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 213–228 (2020)Google Scholar
  15. 15.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2016)Google Scholar
  16. 16.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46475-6_43CrossRefGoogle Scholar
  17. 17.
    Li, C.T., Siu, W.C., Liu, Z.S., Wang, L.W., Lun, D.P.K.: DeepGIN: deep generative inpainting network for extreme image inpainting. In: European Conference on Computer Vision Workshops (2020)Google Scholar
  18. 18.
    Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 702–716. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46487-9_43CrossRefGoogle Scholar
  19. 19.
    Lim, J.H., Ye, J.C.: Geometric GAN. arXiv preprint arXiv:1705.02894 (2017)
  20. 20.
    Nazeri, K., Ng, E., Joseph, T., Qureshi, F., Ebrahimi, M.: EdgeConnect: structure guided image inpainting using edge prediction. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2019Google Scholar
  21. 21.
    Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: EdgeConnect: generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212 (2019)
  22. 22.
    Ntavelis, E., Romero, A., Kastanis, I., Van Gool, L., Timofte, R.: Sesame: semantic editing of scenes by adding, manipulating or erasing objects. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)Google Scholar
  23. 23.
    Ntavelis, E., Romero, A., Bigdeli, S.A., Timofte, R., et al.: AIM 2020 challenge on image extreme inpainting. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 716–741 (2020)Google Scholar
  24. 24.
    Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR, pp. 2337–2346 (2019)Google Scholar
  25. 25.
    Shaker, N., Liapis, A., Togelius, J., Lopes, R., Bidarra, R.: Constructive generation methods for dungeons and levels. Procedural Content Generation in Games. CSCS, pp. 31–55. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-42716-4_3CrossRefGoogle Scholar
  26. 26.
    Son, S., Lee, J., Nah, S., Timofte, R., Lee, K.M., et al.: AIM 2020 challenge on video temporal super-resolution. In: European Conference on Computer Vision Workshops (2020)Google Scholar
  27. 27.
    Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-11021-5_5CrossRefGoogle Scholar
  28. 28.
    Wei, P., Lu, H., Timofte, R., Lin, L., Zuo, W., et al.: AIM 2020 challenge on real image super-resolution : methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 392–422 (2020)Google Scholar
  29. 29.
    Xiangli, Y., Deng, Y., Dai, B., Loy, C.C., Lin, D.: Real or not real, that is the question. In: ICLR (2020)Google Scholar
  30. 30.
    Xu, D., Chu, Y., Sun, Q.: Moire pattern removal via attentive fractal network. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020Google Scholar
  31. 31.
    Yi, Z., Tang, Q., Azizi, S., Jang, D., Xu, Z.: Contextual residual aggregation for ultra high-resolution image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7508–7517 (2020)Google Scholar
  32. 32.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)Google Scholar
  33. 33.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4471–4480 (2019)Google Scholar
  34. 34.
    Zeng, Y., Lin, Z., Yang, J., Zhang, J., Shechtman, E., Lu, H.: High-resolution image inpainting with iterative confidence feedback and guided upsampling. In: European Conference on Computer Vision. Springer (2020)Google Scholar
  35. 35.
    Zhang, K., Danelljan, M., Li, Y., Timofte, R., et al.: AIM 2020 challenge on efficient super-resolution: Methods and results. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020 Workshops. LNCS, vol. 12537, pp. 5–40 (2020)Google Scholar
  36. 36.
    Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2017)CrossRefGoogle Scholar
  37. 37.
    Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633–641 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Computer Vision LabETH ZürichZürichSwitzerland
  2. 2.CSEMNeuchâtelSwitzerland
  3. 3.School of Electronic EngineeringXidian UniversityXi’anChina
  4. 4.Image and Video Pattern Recognition Laboratory, School of Electrical and Electronic EngineeringYonsei UniversitySeoulSouth Korea
  5. 5.Department of Computer Vision (VIS)Baidu Inc.BeijingChina
  6. 6.Samsung R&D Institute China-Beijing (SRC-Beijing)BeijingChina
  7. 7.Dalian University of TechnologyDalianChina
  8. 8.AdobeSan JoseUSA
  9. 9.Rensselaer Polytechnic InstituteTroyUSA
  10. 10.Peking UniversityBeijingChina
  11. 11.Computer Vision and Image Processing (CVIP) LabGachon UniversitySeongnamSouth Korea
  12. 12.Centre for Multimedia Signal Processing, Department of Electronic and Information EngineeringThe Hong Kong Polytechnic UniversityHong KongChina
  13. 13.Indian Institute of Technology MadrasChennaiIndia
  14. 14.BITS PilaniPilaniIndia
  15. 15.MNIT JaipurJaipurIndia

Personalised recommendations