The Visual Computer

, Volume 34, Issue 6–8, pp 1077–1085 | Cite as

Learning to restore deteriorated line drawing

  • Kazuma Sasaki
  • Satoshi Iizuka
  • Edgar Simo-Serra
  • Hiroshi Ishikawa
Original Article


We propose a fully automatic approach to restore aged old line drawings. We decompose the task into two subtasks: the line extraction subtask, which aims to extract line fragments and remove the paper texture background, and the restoration subtask, which fills in possible gaps and deterioration of the lines to produce a clean line drawing. Our approach is based on a convolutional neural network that consists of two sub-networks corresponding to the two subtasks. They are trained as part of a single framework in an end-to-end fashion. We also introduce a new dataset consisting of manually annotated sketches by Leonardo da Vinci which, in combination with a synthetic data generation approach, allows training the network to restore deteriorated line drawings. We evaluate our method on challenging 500-year-old sketches and compare with existing approaches with a user study, in which it is found that our approach is preferred 72.7% of the time.


Image restoration Line drawings Image manipulation Convolutional neural network 



This work was partially supported by JST ACT-I Grant No. JPMJPR16U3, JST ACT-I, Grant No. JPMJPR16UD, and JST CREST Grant No. JPMJCR14D1.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interests.


  1. 1.
    Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D.B.: PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans. Graph. (Proc. of SIGGRAPH 2009) 28(3), 24:1–24:11 (2009)Google Scholar
  2. 2.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)Google Scholar
  3. 3.
    Chiang, J.Y., Tue, S., Leu, Y.: A new algorithm for line image vectorization. Pattern Recognit. 31(10), 1541–1549 (1998)CrossRefGoogle Scholar
  4. 4.
    Cugini, U., Ferri, G., Mussio, P., Protti, M.: Pattern-directed restoration and vectorization of digitized engineering drawings. Comput. Graph. 8(4), 337–350 (1984)CrossRefGoogle Scholar
  5. 5.
    Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Darabi, S., Shechtman, E., Barnes, C., Goldman, D.B., Sen, P.: Image melding: combining inconsistent images using patch-based synthesis. ACM Trans. Graph. (Proc. SIGGRAPH 2012) 31(4), 82:1–82:10 (2012)Google Scholar
  7. 7.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision, pp. 184–199. Springer, Berlin (2014)Google Scholar
  8. 8.
    Favreau, J.D., Lafarge, F., Bousseau, A.: Fidelity vs. simplicity: a global approach to line drawing vectorization. ACM Trans. Graph. (TOG) 35(4), 120 (2016)CrossRefGoogle Scholar
  9. 9.
    Gu, S., Zuo, W., Xie, Q., Meng, D., Feng, X., Zhang, L.: Convolutional sparse coding for image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1823–1831 (2015)Google Scholar
  10. 10.
    Han, C.C., Fan, K.C.: Skeleton generation of engineering drawings via contour matching. Pattern Recognit. 27(2), 261–275 (1994)CrossRefGoogle Scholar
  11. 11.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (Proc. SIGGRAPH 2017) 36(4), 107:1–107:14 (2017)Google Scholar
  12. 12.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning (2015)Google Scholar
  13. 13.
    Jain, V., Seung, S.: Natural image denoising with convolutional networks. In: Advances in Neural Information Processing Systems, pp. 769–776 (2009)Google Scholar
  14. 14.
    Janssen, R.D., Vossepoel, A.M.: Adaptive vectorization of line drawing images. Comput. Vis. Image Underst. 65(1), 38–56 (1997)CrossRefGoogle Scholar
  15. 15.
    Komodakis, N., Tziritas, G.: Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Trans. Image Process. 16(11), 2649–2661 (2007)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Li, C., Liu, X., Wong, T.T.: Deep extraction of manga structural lines. ACM Trans. Graph. 36(4), 117:1–117:12 (2017)Google Scholar
  17. 17.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  18. 18.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder–decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810 (2016)Google Scholar
  19. 19.
    Milanfar, P.: A tour of modern image filtering: new insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30(1), 106–128 (2013)CrossRefGoogle Scholar
  20. 20.
    Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Model. Simul. 4(2), 460–489 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.: Context encoders: feature learning by inpainting. In: Computer Vision and Pattern Recognition (2016)Google Scholar
  22. 22.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1–4), 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Sasaki, K., Iizuka, S., Simo-Serra, E., Ishikawa, H.: Joint gap detection and inpainting of line drawings. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  24. 24.
    Simakov, D., Caspi, Y., Shechtman, E., Irani, M.: Summarizing visual data using bidirectional similarity. In: Computer Vision and Pattern Recognition, pp. 1–8 (2008)Google Scholar
  25. 25.
    Simo-Serra, E., Iizuka, S., Ishikawa, H.: Mastering sketching: adversarial augmentation for structured prediction. Trans. Graph. (TOG) 37(1), 11 (2018)Google Scholar
  26. 26.
    Simo-Serra, E., Iizuka, S., Sasaki, K., Ishikawa, H.: Learning to simplify: fully convolutional networks for rough sketch cleanup. ACM Trans. Graph. (in Proc. of SIGGRAPH 2016) 35(4), 121 (2016)Google Scholar
  27. 27.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)Google Scholar
  28. 28.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)Google Scholar
  29. 29.
    Wang, Z., Yang, Y., Wang, Z., Chang, S., Yang, J., Huang, T.S.: Learning super-resolution jointly from external and internal examples. IEEE Trans. Image Process. 24(11), 4359–4371 (2015)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. IEEE Trans. Pattern Anal. Mach. Intell. 29(3), 463–476 (2007)CrossRefGoogle Scholar
  31. 31.
    Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, pp. 341–349 (2012)Google Scholar
  32. 32.
    Zeiler, M.D.: Adadelta: An Adaptive Learning Rate Method. arXiv preprint arXiv:1212.5701 (2012)

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Waseda UniversityTokyoJapan

Personalised recommendations