Machine Vision and Applications

, Volume 30, Issue 7–8, pp 1229–1241 | Cite as

Multi-scale convolutional neural network for pixel-wise reconstruction of Van Gogh’s drawings

  • Y. ZengEmail author
  • J. C. A. van der Lubbe
  • M. Loog
Original Paper


This paper investigates the reconstruction of Van Gogh’s drawings which have been degraded in the course of time due to aging problems, like ink fading and discoloration. Learning to predict the past and original appearances of degraded drawings can help to envisage how the artist’s work may have looked at the time of creation. In this paper, we use reproductions as reference information for the past appearances of drawings and consider the reconstruction of drawings as a pixel-wise prediction problem. We present an approach to automatically predict the past appearances of drawings. This approach brings together methods from multi-resolution image analysis and deep convolutional neural networks (CNNs) for addressing the task of pixel-wise prediction. Our experiments first investigate how scale affects prediction performance of the proposed multi-scale CNN framework and then demonstrate the reconstruction capability of the multi-scale CNN framework. The results demonstrate that the predictive reconstruction of degraded images is a feasible endeavor.


Convolutional neural networks Image reconstruction Pixel-wise prediction Van Gogh’s drawings 



This research is a part of the REVIGO Project, supported by the Netherlands Organization for Scientific Research (NWO; Grant 323.54.004) in the context of the Science4Arts research program.


  1. 1.
    Vellekoop, M., Jansen, L., Geldof, M., Hendriks, E., Tagle, A.: Van Goghs Studio Practice. Van Gogh Museum, Brussels (2013)Google Scholar
  2. 2.
    Berezhnoy, I.E., Postma, E.O., van den Herik, H.J.: Automatic extraction of brushstroke orientation from paintings. Mach. Vis. Appl. 20(1), 1–9 (2009)CrossRefGoogle Scholar
  3. 3.
    Cornelis, B., Yang, H., Goodfriend, A., Ocon, N., Lu, J., Daubechies, I.: Removal of canvas patterns in digital acquisitions of paintings. IEEE Trans. Image Process. 26(1), 160–171 (2017)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Yan, C., Xie, H., Chen, J., Zha, Z., Hao, X., Zhang, Y., Dai, Q.: A fast uyghur text detector for complex background images. IEEE Trans. Multimed. 20(12), 3389–3398 (2018)CrossRefGoogle Scholar
  5. 5.
    Zeng, Y., Tang, J., van der Lubbe, J., Loog, M.: Learning algorithms for digital reconstruction of Van Gogh’s drawings. In: 6th International Conference, EuroMed 2016, Cyprus, vol. 10058, pp. 322–333 (2016)CrossRefGoogle Scholar
  6. 6.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 1520–1528 (2015) (to appear)Google Scholar
  7. 7.
    Pappas, M., Pitas, I.: Digital color restoration of old paintings. IEEE Trans. Image Process. 9(2), 291–294 (2000)CrossRefGoogle Scholar
  8. 8.
    Pei, S., Zeng, Y., Chang, C.: Virtual restoration of ancient Chinese paintings using color contrast enhancement and lacuna texture synthesis. IEEE Trans. Image Process. 13(3), 416–429 (2004)CrossRefGoogle Scholar
  9. 9.
    Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.: Image analogies. In: SIGGRAPH ’01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, pp. 327–340. ACM Press (2001)Google Scholar
  10. 10.
    Laffont, P., Ren, Z., Tao, X., Qian, C., Hays, J.: Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33(4), 149 (2014)CrossRefGoogle Scholar
  11. 11.
    Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32(6), 200:1–200:11 (2013)CrossRefGoogle Scholar
  12. 12.
    Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)CrossRefGoogle Scholar
  13. 13.
    Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: CVPR, pp. 769–777. IEEE Computer Society (2015)Google Scholar
  14. 14.
    Nah, S., Kim, T.H., Lee, K.M.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 257–265. IEEE Computer Society (2017)Google Scholar
  15. 15.
    Chen, Y., Yu, W., Pock, T.: On learning optimized reaction diffusion processes for effective image restoration. In: CVPR, pp. 5261–5269. IEEE Computer Society (2015)Google Scholar
  16. 16.
    Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 5882–5891(2017)Google Scholar
  17. 17.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  18. 18.
    Wang, X., Yu, K., Dong, C., Loy, C.C.: Recovering realistic texture in image super-resolution by deep spatial feature transform. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 606–615 (2018)Google Scholar
  19. 19.
    Chaudhury, S., Roy, H.: Can fully convolutional networks perform well for general image restoration problems? In: MVA, pp. 254–257. IEEE (2017)Google Scholar
  20. 20.
    Kim, J., Lee, J.K., Lee, M.K: Accurate image super-resolution using very deep convolutional networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016, pp. 1646–1654 (2016)Google Scholar
  21. 21.
    Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)Google Scholar
  22. 22.
    Mathieu, M., Couprie, C., LeCun, Y.: Deep multi-scale video prediction beyond mean square error. CoRR (2015). arXiv:1511.05440
  23. 23.
    van Noord, N., Postma, E.O.: Learning scale-variant and scale-invariant features for deep image classification. Pattern Recognit. 61, 583–592 (2017)CrossRefGoogle Scholar
  24. 24.
    Yoo, D., Park, S., Lee, J., So Kweon, I.: Multi-scale pyramid pooling for deep convolutional representation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2015)Google Scholar
  25. 25.
    Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Bach, F.R., Blei, D.M. (eds.) ICML, JMLR Workshop and Conference Proceedings, vol. 37, pp. 448–456. (2015)Google Scholar
  26. 26.
    Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: Frnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814. Omni Press, Madison (2010)Google Scholar
  27. 27.
    Jian, B., Vemuri, B.C.: Robust point set registration using Gaussian mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1633–1645 (2011)CrossRefGoogle Scholar
  28. 28.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European Conference on Computer Vision (2016)Google Scholar
  29. 29.
    Xu, L., Ren, S.J., Liu, C.E., Jia, J.: Deep convolutional neural network for image deconvolution. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14, vol. 1, pp. 1790–1798. MIT Press, Cambridge (2014)Google Scholar
  30. 30.
    Nah, S., Kim, T., Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 257–265. IEEE Computer Society, Los Alamitos (2017)Google Scholar
  31. 31.
    Neevel, H., van Bommel, M., Vellekoop, M., Lingbeek, N.: From purple to brown—analysis of Van Gogh’s drawing inks; non-invasive identification of inks on Van Gogh’s drawings and letters. In: Proceedings of the 15th Triennial Conference, vol. 15, pp. 278–284 (2008)Google Scholar
  32. 32.
    Geerts, J.: Light damage on paper objects with coloured media; towards a differential sensitivity classification. MA thesis (2010)Google Scholar
  33. 33.
    Conover, D.M., Delaney, J.K., Loew, M.H.: Automatic registration and mosaicking of technical images of old master paintings. Appl. Phys. A 119(4), 1567–1575 (2015)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Southern University of Science and TechnologyShenzhenChina
  2. 2.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations