Advertisement

CNN-Based Stereoscopic Image Inpainting

  • Shen ChenEmail author
  • Wei MaEmail author
  • Yue QinEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11903)

Abstract

CNN has proved powerful in many tasks, including single image inpainting. The paper presents an end-to-end network for stereoscopic image inpainting. The proposed network is composed of two encoders for independent feature extraction of a pair of stereo images with missing regions, a feature fusion module for stereo coherent structure prediction, and two decoders to generate a pair of completed images. In order to train the model, besides a reconstruction and an adversarial loss for content recovery, a local consistency loss is defined to constrain stereo coherent detail prediction. Moreover, we present a transfer-learning based training strategy to solve the issue of stereoscopic data scarcity. To the best of our knowledge, we are the first to solve the stereoscopic inpainting problem in the framework of CNN. Compared to traditional stereoscopic inpainting and available CNN-based single image inpainting (repairing stereo views one by one) methods, our network generates results of higher image quality and stereo consistency.

Keywords

Stereoscopic vision Image inpainting Convolutional Neural Network 

References

  1. 1.
    Baek, S.H., Choi, I., Kim, M.H.: Multiview image completion with space structure propagation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 488–496 (2016)Google Scholar
  2. 2.
    Chang, C.H., Liang, C.K., Chuang, Y.Y.: Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans. Multimedia 13(4), 589–601 (2011)CrossRefGoogle Scholar
  3. 3.
    Chen, D., Liao, J., Yuan, L., Yu, N., Hua, G.: Coherent online video style transfer. In: IEEE International Conference on Computer Vision, pp. 1114–1123 (2017)Google Scholar
  4. 4.
    Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: Stereoscopic neural style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition (2018)Google Scholar
  5. 5.
    Demir, U., Ünal, G.B.: Deep stacked networks with residual polishing for image inpainting. arXiv:1801.00289 (2017)
  6. 6.
    Domingos, P.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012)CrossRefGoogle Scholar
  7. 7.
    Dosovitskiy, A., Springenberg, J.T., Brox, T.: Learning to generate chairs with convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1538–1546 (2015)Google Scholar
  8. 8.
    Egnal, G.: Mutual information as a stereo correspondence measure. Technical reports (2000)Google Scholar
  9. 9.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Robot. Res. 32(11), 1231–1237 (2013)CrossRefGoogle Scholar
  10. 10.
    Gong, X., Huang, H., Ma, L., Shen, F., Liu, W.: Neural stereoscopic image style transfer. arXiv:1802.09985 (2018)CrossRefGoogle Scholar
  11. 11.
    Goodfellow, I.J., et al.: Generative adversarial nets. In: International Conference on Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  12. 12.
    Hirschmüller, H., Scharstein, D.: Evaluation of cost functions for stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  13. 13.
    Horé, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: International Conference on Pattern Recognition, pp. 2366–2369 (2010)Google Scholar
  14. 14.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. 36(4), 107:1–107:14 (2017)CrossRefGoogle Scholar
  15. 15.
    Lo, W.Y., van Baar, J., Knaus, C., Zwicker, M., Gross, M.: Stereoscopic 3D copy & paste. ACM Trans. Graph. 29(6), 147:1–147:10 (2010)CrossRefGoogle Scholar
  16. 16.
    Luo, S.J., Sun, Y.T., Shen, I.C., Chen, B.Y., Chuang, Y.Y.: Geometrically consistent stereoscopic image editing using patch-based synthesis. IEEE Trans. Visual Comput. Graph. 21(1), 56–67 (2015)CrossRefGoogle Scholar
  17. 17.
    Ma, W., Qin, Y., Xu, S., Zhang, X.: Interactive stereo image segmentation via adaptive prior selection. Multimedia Tools Appl. 77, 28709–28724 (2018)CrossRefGoogle Scholar
  18. 18.
    Ma, W., Qin, Y., Yang, L., Xu, S., Zhang, X.: Interactive stereo image segmentation with RGB-D hybrid constraints. IEEE Signal Process. Lett. 23, 1533–1537 (2016)CrossRefGoogle Scholar
  19. 19.
    Ma, W., Yang, L., Zhang, Y.L., Duan, L.: Fast interactive stereo image segmentation. Multimedia Tools Appl. 75, 10935–10948 (2015)CrossRefGoogle Scholar
  20. 20.
    Mayer, N., et al.: A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4040–4048 (2016)Google Scholar
  21. 21.
    Morse, B.S., Howard, J., Cohen, S., Price, B.L.: PatchMatch-based content completion of stereo image pairs. In: International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission, pp. 555–562 (2012)Google Scholar
  22. 22.
    Mu, T.J., Wang, J.H., Du, S.P., Hu, S.: Stereoscopic image completion and depth recovery. Vis. Comput. 30(6–8), 833–843 (2014)CrossRefGoogle Scholar
  23. 23.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)Google Scholar
  24. 24.
    Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)Google Scholar
  25. 25.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Thaskani, S., Karande, S.S., Lodha, S.: Multi-view image inpainting with sparse representations. In: IEEE International Conference on Image Processing, pp. 1414–1418 (2015)Google Scholar
  27. 27.
    Wang, L., Jin, H., Yang, R., Gong, M.: Stereoscopic inpainting: joint color and depth completion from stereo images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  28. 28.
    Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: towards realistic high-resolution image blending. arXiv:1703.07195 (2017)
  29. 29.
    Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. IEEE Trans. Image Process. 19(5), 1153–11657 (2010)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4076–4084 (2017)Google Scholar
  31. 31.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 801, pp. 151–158. Springer, Heidelberg (1994).  https://doi.org/10.1007/BFb0028345CrossRefGoogle Scholar
  32. 32.
    Zbontar, J., LeCun, Y.: Computing the stereo matching cost with a convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1592–1599 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Beijing University of TechnologyBeijingChina

Personalised recommendations