Digital Heritage Reconstruction Using Deep Learning-Based Super-Resolution

  • Prathmesh R. Madhu
  • Manjunath V. Joshi


Heritage sites and archival monuments have a great cultural significance. However, they suffer degradation due to several reasons. As a result, in order to preserve the cultural heritage, one has seen increased interest in research on digitally restoring the photographs of vandalized monuments. One may think of recreating the historical monuments by super-resolving the heritage images, an algorithmic approach to increase the spatial resolution of an image. This chapter presents a single image super-resolution (SR) method based on deep learning to obtain higher resolution photographs of the digitally reconstructed monuments. The resulting images can serve as the input to walkthrough systems. Given a low spatial resolution test image and a database consisting of low and high spatial resolution (LR-HR) images, we obtain super-resolution for the test image. We use the idea proposed in Dong et al. (Computer Vision–ECCV 2014, Springer, 2014, [5]) to represent the mapping between LR and HR images by using a deep convolutional neural network (CNN). CNN filters are learned by standard backpropagation and stochastic gradient descent method. The novelty of our approach lies in the elimination of interpolation during the training phase. Our method directly learns the end-to-end mapping between LR and HR images. The advantage of our approach is that once the network is trained for a magnification factor of 2, the learned parameters can be used to obtain SR for higher magnification factors also. We demonstrate the effectiveness of the proposed approach by conducting experiments using the images of heritage monuments as well as natural scene. Our results are compared with the standard interpolation technique and existing learning-based approaches. Visual and quantitative comparisons confirm the effectiveness of the proposed method.


Super-resolution Deep convolutional neural networks Deep learning Back propagation Stochastic gradient descent 



The authors would like to thank NVIDIA Corporation for providing the TITAN X GPU for the academic research. The authors are also immensely grateful to the reviewers of the book for their comments on the earlier versions of the manuscript. They are also thankful to their colleagues Dr. Milind G. Padalkar, Meet H. Soni, Ketul D. Parikh and Surabhi D. Sohoney for sharing their pearls of wisdom with them during the course of this research.


  1. 1.
    Baker, S., Kanade, T.: Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 24(9), 1167–1183 (2002)CrossRefGoogle Scholar
  2. 2.
    Bose, N., Kim, H., Valenzuela, H.: Recursive implementation of total least squares algorithm for image reconstruction from noisy, undersampled multiframes. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1993. ICASSP-93, vol. 5, pp. 269–272. IEEE (1993)Google Scholar
  3. 3.
    Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising with multi-layer perceptrons, part 1: comparison with existing algorithms and with bounds. arXiv:1211.1544 (2012)
  4. 4.
    Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X.: Deep network cascade for image super-resolution. In: Computer Vision–ECCV 2014, pp. 49–64. Springer (2014)Google Scholar
  5. 5.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision–ECCV 2014, pp. 184–199. Springer (2014)Google Scholar
  6. 6.
    Farsiu, S., Robinson, D., Elad, M., Milanfar, P.: Advances and challenges in super-resolution. Int. J. Imaging Syst. Technol. 14(2), 47–57 (2004)CrossRefGoogle Scholar
  7. 7.
    Fattal, R.: Image upsampling via imposed edge statistics. In: ACM Transactions on Graphics (TOG), vol. 26, p. 95. ACM (2007)CrossRefGoogle Scholar
  8. 8.
    Freedman, G., Fattal, R.: Image and video upscaling from local self-examples. ACM Trans. Graph. (TOG) 30(2), 12 (2011)CrossRefGoogle Scholar
  9. 9.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graph. Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  10. 10.
    Gajjar, P.P., Joshi, M.V.: New learning based super-resolution: use of DWT and IGMRF prior. IEEE Trans. Image Process. 19(5), 1201–1213 (2010)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 349–356. IEEE (2009)Google Scholar
  12. 12.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Kim, K.I., Kwon, Y.: Example-based learning for single-image super-resolution. In: Pattern Recognition, pp. 456–465. Springer (2008)Google Scholar
  14. 14.
    Kim, S.P., Su, W.Y.: Recursive high-resolution reconstruction of blurred multiframe images. IEEE Trans. Image Process. 2(4), 534–539 (1993)CrossRefGoogle Scholar
  15. 15.
    Li, F.F., Karpathy, A., Johnson, J.: Cs231n: convolutional neural networks for visual recognition (2016). Accessed 28 June 2016Google Scholar
  16. 16.
    Lin, Z., Shum, H.Y.: Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 83–97 (2004)CrossRefGoogle Scholar
  17. 17.
    Park, S.C., Park, M.K., Kang, M.G.: Super-resolution image reconstruction: a technical overview. IEEE Signal Process. Mag. 20(3), 21–36 (2003)CrossRefGoogle Scholar
  18. 18.
    Protter, M., Elad, M., Takeda, H., Milanfar, P.: Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Trans. Image Process. 18(1), 36–51 (2009)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Rhee, S., Kang, M.G.: Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Opt. Eng. 38(8), 1348–1356 (1999)CrossRefGoogle Scholar
  20. 20.
    Sun, J., Sun, J., Xu, Z., Shum, H.Y.: Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  21. 21.
    Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: The IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  22. 22.
    Tsai, R., Huang, T.S.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1(2), 317–339 (1984)Google Scholar
  23. 23.
    Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)Google Scholar
  24. 24.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Dhirubhai Ambani Institute of Information and Communication TechnologyGandhinagarIndia

Personalised recommendations