Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings

Abstract

This paper investigates the reconstruction of Van Gogh’s drawings which have been degraded in the course of time due to aging problems, like ink fading and discoloration. Learning to predict the past and original appearances of degraded drawings can help to envisage how the artist’s work may have looked at the time of creation. In this paper, we use reproductions as reference information for the past appearances of drawings and consider the reconstruction of drawings as a pixel-wise prediction problem. We present an approach to automatically predict the past appearances of drawings. This approach brings together methods from multi-resolution image analysis and deep convolutional neural networks (CNNs) for addressing the task of pixel-wise prediction. Our experiments first investigate how scale affects prediction performance of the proposed multi-scale CNN framework and then demonstrate the reconstruction capability of the multi-scale CNN framework. The results demonstrate that the predictive reconstruction of degraded images is a feasible endeavor.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

References

  1. 1.

    Vellekoop, M., Jansen, L., Geldof, M., Hendriks, E., Tagle, A.: Van Goghs Studio Practice. Van Gogh Museum, Brussels (2013)

    Google Scholar 

  2. 2.

    Berezhnoy, I.E., Postma, E.O., van den Herik, H.J.: Automatic extraction of brushstroke orientation from paintings. Mach. Vis. Appl. 20(1), 1–9 (2009)

    Article  Google Scholar 

  3. 3.

    Cornelis, B., Yang, H., Goodfriend, A., Ocon, N., Lu, J., Daubechies, I.: Removal of canvas patterns in digital acquisitions of paintings. IEEE Trans. Image Process. 26(1), 160–171 (2017)

    MathSciNet  Article  Google Scholar 

  4. 4.

    Yan, C., Xie, H., Chen, J., Zha, Z., Hao, X., Zhang, Y., Dai, Q.: A fast uyghur text detector for complex background images. IEEE Trans. Multimed. 20(12), 3389–3398 (2018)

    Article  Google Scholar 

  5. 5.

    Zeng, Y., Tang, J., van der Lubbe, J., Loog, M.: Learning algorithms for digital reconstruction of Van Gogh’s drawings. In: 6th International Conference, EuroMed 2016, Cyprus, vol. 10058, pp. 322–333 (2016)

    Google Scholar 

  6. 6.

    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 1520–1528 (2015) (to appear)

  7. 7.

    Pappas, M., Pitas, I.: Digital color restoration of old paintings. IEEE Trans. Image Process. 9(2), 291–294 (2000)

    Article  Google Scholar 

  8. 8.

    Pei, S., Zeng, Y., Chang, C.: Virtual restoration of ancient Chinese paintings using color contrast enhancement and lacuna texture synthesis. IEEE Trans. Image Process. 13(3), 416–429 (2004)

    Article  Google Scholar 

  9. 9.

    Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH ’01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, pp. 327–340. ACM Press (2001)

  10. 10.

    Laffont, P., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33(4), 149 (2014)

    Article  Google Scholar 

  11. 11.

    Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32(6), 200:1–200:11 (2013)

    Article  Google Scholar 

  12. 12.

    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)

    Article  Google Scholar 

  13. 13.

    Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: CVPR, pp. 769–777. IEEE Computer Society (2015)

  14. 14.

    Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 257–265. IEEE Computer Society (2017)

  15. 15.

    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: CVPR, pp. 5261–5269. IEEE Computer Society (2015)

  16. 16.

    Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5882–5891(2017)

  17. 17.

    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  18. 18.

    Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 606–615 (2018)

  19. 19.

    Chaudhury, S., Roy, H.: Can fully convolutional networks perform well for general image restoration problems? In: MVA, pp. 254–257. IEEE (2017)

  20. 20.

    Kim, J., Lee, J.K., Lee, M.K: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 1646–1654 (2016)

  21. 21.

    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)

  22. 22.

    Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. CoRR (2015). arXiv:1511.05440

  23. 23.

    van Noord, N., Postma, E.O.: Learning scale-variant and scale-invariant features for deep image classification. Pattern Recognit. 61, 583–592 (2017)

    Article  Google Scholar 

  24. 24.

    Yoo, D., Park, S., Lee, J., So Kweon, I.: Multi-scale pyramid pooling for deep convolutional representation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2015)

  25. 25.

    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F.R., Blei, D.M. (eds.) ICML, JMLR Workshop and Conference Proceedings, vol. 37, pp. 448–456. JMLR.org (2015)

  26. 26.

    Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: Frnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814. Omni Press, Madison (2010)

    Google Scholar 

  27. 27.

    Jian, B., Vemuri, B.C.: Robust point set registration using Gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2011)

    Article  Google Scholar 

  28. 28.

    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)

  29. 29.

    Xu, L., Ren, S.J., Liu, C.E., Jia, J.: Deep convolutional neural network for image deconvolution. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14, vol. 1, pp. 1790–1798. MIT Press, Cambridge (2014)

  30. 30.

    Nah, S., Kim, T., Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265. IEEE Computer Society, Los Alamitos (2017)

  31. 31.

    Neevel, H., van Bommel, M., Vellekoop, M., Lingbeek, N.: From purple to brown—analysis of Van Gogh’s drawing inks; non-invasive identification of inks on Van Gogh’s drawings and letters. In: Proceedings of the 15th Triennial Conference, vol. 15, pp. 278–284 (2008)

  32. 32.

    Geerts, J.: Light damage on paper objects with coloured media; towards a differential sensitivity classification. MA thesis (2010)

  33. 33.

    Conover, D.M., Delaney, J.K., Loew, M.H.: Automatic registration and mosaicking of technical images of old master paintings. Appl. Phys. A 119(4), 1567–1575 (2015)

    Article  Google Scholar 

Download references

Acknowledgements

This research is a part of the REVIGO Project, supported by the Netherlands Organization for Scientific Research (NWO; Grant 323.54.004) in the context of the Science4Arts research program.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Y. Zeng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zeng, Y., van der Lubbe, J.C.A. & Loog, M. Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings. Machine Vision and Applications 30, 1229–1241 (2019). https://doi.org/10.1007/s00138-019-01047-3

Download citation

Keywords

  • Convolutional neural networks
  • Image reconstruction
  • Pixel-wise prediction
  • Van Gogh’s drawings