Architecture Search for Image Inpainting

  • Yaoman LiEmail author
  • Irwin King
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11554)


Neural Architecture Search (NAS) shows the ability to automate the architecture engineering for specific tasks recently which is extremely promising. Many published works apply reinforcement learning or evolutionary algorithm to design the neural architecture for image classification and achieve state-of-the-art performance. However, using NAS to perform other challenging tasks, such as inpainting irregular regions in an image, has not been explored yet. The target of image inpainting is to generate plausible image regions to fill the missing regions in the original image. It has been widely used in many applications. In this paper, we are interested in applying neural architecture search methods to image inpainting tasks. We propose to use reinforcement learning to automatically design the network architecture. Our method can efficiently explore new network structure based on existing architecture. The experiment result demonstrates that the proposed method can design an efficient and high-quality architecture for image inpainting.


Reinforcement learning Neural architecture search Image inpainting Partial convolution AutoML U-Net 



The work described in this paper was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14208815 of the General Research Fund).


  1. 1.
    Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Trans. Neural Netw. 5(1), 54–65 (1994)Google Scholar
  2. 2.
    Baker, B., Gupta, O., Naik, N., Raskar, R.: Designing neural network architectures using reinforcement learning. CoRR abs/1611.02167 (2016)Google Scholar
  3. 3.
    Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., Verdera, J.: Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. Image Process. 10(8), 1200–1211 (2001)Google Scholar
  4. 4.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28(3), 24:1–24:11 (2009)Google Scholar
  5. 5.
    Bertalmío, M., Sapiro, G., Caselles, V., Ballester, C.: Image in painting. In: SIGGRAPH, pp. 417–424. ACM (2000)Google Scholar
  6. 6.
    Cai, H., Chen, T., Zhang, W., Yu, Y., Wang, J.: Efficient architecture search by network transformation. In: AAAI, pp. 2787–2794. AAAI Press (2018)Google Scholar
  7. 7.
    Cai, H., Yang, J., Zhang, W., Han, S., Yu, Y.: Path-level network transformation for efficient architecture search. In: JMLR Workshop and Conference Proceedings, ICML, vol. 80, pp. 677–686. (2018)Google Scholar
  8. 8.
    Chen, T., Goodfellow, I.J., Shlens, J.: Net2Net: accelerating learning via knowledge transfer. In: ICLR (2016)Google Scholar
  9. 9.
    Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: SIGGRAPH, pp. 341–346. ACM (2001)Google Scholar
  10. 10.
    Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: ICCV, pp. 1033–1038 (1999)Google Scholar
  11. 11.
    Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: from architectures to learning. Evol. Intell. 1(1), 47–62 (2008)Google Scholar
  12. 12.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 107:1–107:14 (2017)Google Scholar
  13. 13.
    Köhler, R., Schuler, C., Schölkopf, B., Harmeling, S.: Mask-specific inpainting with deep neural networks. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 523–534. Springer, Cham (2014). Scholar
  14. 14.
    Liu, C., et al.: Progressive neural architecture search. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 19–35. Springer, Cham (2018). Scholar
  15. 15.
    Liu, G., Reda, F.A., Shih, K.J., Wang, T.-C., Tao, A., Catanzaro, B.: Image inpainting for irregular holes using partial convolutions. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 89–105. Springer, Cham (2018). Scholar
  16. 16.
    Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. In: ICLR (2018)Google Scholar
  17. 17.
    Pham, H., Guan, M.Y., Zoph, B., Le, Q.V., Dean, J.: Efficient neural architecture search via parameter sharing. In: JMLR Workshop and Conference Proceedings, ICML, vol. 80, pp. 4092–4101. (2018)Google Scholar
  18. 18.
    Real, E., et al.: Large-scale evolution of image classifiers. In: Proceedings of Machine Learning Research, ICML, vol. 70, pp. 2902–2911. PMLR (2017)Google Scholar
  19. 19.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). Scholar
  20. 20.
    Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)Google Scholar
  21. 21.
    Suganuma, M., Shirakawa, S., Nagao, T.: A genetic programming approach to designing convolutional neural network architectures. In: IJCAI, pp. 5369–5373. (2018)Google Scholar
  22. 22.
    Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. In: ACL (1), pp. 1556–1566. The Association for Computer Linguistics (2015)Google Scholar
  23. 23.
    Xu, L., Ren, J.S.J., Liu, C., Jia, J.: Deep convolutional neural network for image deconvolution. In: NIPS, pp. 1790–1798 (2014)Google Scholar
  24. 24.
    Yan, Z., Li, X., Li, M., Zuo, W., Shan, S.: Shift-Net: image inpainting via deep feature rearrangement. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 3–19. Springer, Cham (2018). Scholar
  25. 25.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image in painting with contextual attention. In: CVPR, pp. 5505–5514. IEEE Computer Society (2018)Google Scholar
  26. 26.
    Zhong, Z., Yan, J., Wu, W., Shao, J., Liu, C.: Practical block-wise neural network architecture generation. In: CVPR, pp. 2423–2432. IEEE Computer Society (2018)Google Scholar
  27. 27.
    Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. CoRR abs/1611.01578 (2016)Google Scholar
  28. 28.
    Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: CVPR, pp. 8697–8710. IEEE Computer Society (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.The Chinese University of Hong KongShatinHong Kong
  2. 2.Lenovo Group Ltd.BeijingChina

Personalised recommendations