Digital Heritage Reconstruction Using Deep Learning-Based Super-Resolution
Heritage sites and archival monuments have a great cultural significance. However, they suffer degradation due to several reasons. As a result, in order to preserve the cultural heritage, one has seen increased interest in research on digitally restoring the photographs of vandalized monuments. One may think of recreating the historical monuments by super-resolving the heritage images, an algorithmic approach to increase the spatial resolution of an image. This chapter presents a single image super-resolution (SR) method based on deep learning to obtain higher resolution photographs of the digitally reconstructed monuments. The resulting images can serve as the input to walkthrough systems. Given a low spatial resolution test image and a database consisting of low and high spatial resolution (LR-HR) images, we obtain super-resolution for the test image. We use the idea proposed in Dong et al. (Computer Vision–ECCV 2014, Springer, 2014, ) to represent the mapping between LR and HR images by using a deep convolutional neural network (CNN). CNN filters are learned by standard backpropagation and stochastic gradient descent method. The novelty of our approach lies in the elimination of interpolation during the training phase. Our method directly learns the end-to-end mapping between LR and HR images. The advantage of our approach is that once the network is trained for a magnification factor of 2, the learned parameters can be used to obtain SR for higher magnification factors also. We demonstrate the effectiveness of the proposed approach by conducting experiments using the images of heritage monuments as well as natural scene. Our results are compared with the standard interpolation technique and existing learning-based approaches. Visual and quantitative comparisons confirm the effectiveness of the proposed method.
KeywordsSuper-resolution Deep convolutional neural networks Deep learning Back propagation Stochastic gradient descent
The authors would like to thank NVIDIA Corporation for providing the TITAN X GPU for the academic research. The authors are also immensely grateful to the reviewers of the book for their comments on the earlier versions of the manuscript. They are also thankful to their colleagues Dr. Milind G. Padalkar, Meet H. Soni, Ketul D. Parikh and Surabhi D. Sohoney for sharing their pearls of wisdom with them during the course of this research.
- 2.Bose, N., Kim, H., Valenzuela, H.: Recursive implementation of total least squares algorithm for image reconstruction from noisy, undersampled multiframes. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1993. ICASSP-93, vol. 5, pp. 269–272. IEEE (1993)Google Scholar
- 3.Burger, H.C., Schuler, C.J., Harmeling, S.: Image denoising with multi-layer perceptrons, part 1: comparison with existing algorithms and with bounds. arXiv:1211.1544 (2012)
- 4.Cui, Z., Chang, H., Shan, S., Zhong, B., Chen, X.: Deep network cascade for image super-resolution. In: Computer Vision–ECCV 2014, pp. 49–64. Springer (2014)Google Scholar
- 5.Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Computer Vision–ECCV 2014, pp. 184–199. Springer (2014)Google Scholar
- 11.Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 349–356. IEEE (2009)Google Scholar
- 13.Kim, K.I., Kwon, Y.: Example-based learning for single-image super-resolution. In: Pattern Recognition, pp. 456–465. Springer (2008)Google Scholar
- 15.Li, F.F., Karpathy, A., Johnson, J.: Cs231n: convolutional neural networks for visual recognition (2016). Accessed 28 June 2016Google Scholar
- 20.Sun, J., Sun, J., Xu, Z., Shum, H.Y.: Image super-resolution using gradient profile prior. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
- 21.Timofte, R., De Smet, V., Van Gool, L.: Anchored neighborhood regression for fast example-based super-resolution. In: The IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
- 22.Tsai, R., Huang, T.S.: Multiframe image restoration and registration. Adv. Comput. Vis. Image Process. 1(2), 317–339 (1984)Google Scholar
- 23.Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103. ACM (2008)Google Scholar
- 24.Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)Google Scholar