Abstract
Physical artefacts or ancient artworks including manuscripts and photographs at heritage sites are likely to have significantly damaged or broken-down regions. Therefore, digital image restoration is utilised in cultural heritage (CH) preservation and restoration by taking digital photographs of the historical artefacts. Then, numerous image processing techniques including image denoising, image decomposition, image deblurring, image inpainting, etc. can be applied as a post-processing step to improve the quality of these photographs. Of these techniques, image inpainting plays a crucial role in filling missing parts and recovering damaged regions of digital images. This chapter presents several digital image inpainting techniques that can be used to perform inpainting for CH. Also, this chapter presents a new two-stage inpainting method for cultural heritage digital image using Delaunay triangulation-based interpolation and exemplar-based inpainting method. This method, first, assigns colours to the unknown pixels, i.e. the selected region to be inpainted, from the nearest known pixels of the images using Delaunay triangulation-based interpolation. Then, it employs an exemplar-based algorithm to perform the inpainting on the interpolated image. The experimental results showed that our proposed method can produce better results than the stand-alone exemplar-based inpainting method. Moreover, examples of inpainting tasks showed the potential of our proposed method in producing high-quality inpainting results and, at the same time, overcome the limitations of exemplar-based inpainting.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)
Paunwala, C.N.: Image inpainting evolution: a survey. In: Encyclopedia of Image Processing, p. 293. CRC Press, Boca Raton (2018)
Yoo, S., Park, R.H.: Red-eye detection and correction using inpainting in digital photographs. IEEE Trans. Consum. Electron. 55(3), 1006–1014 (2009)
Chan, T.F., Yip, A.M., Park, F.E.: Simultaneous total variation image inpainting and blind deconvolution. Int. J. Imaging Syst. Technol. 15(1), 92–102 (2005)
Jones, B.G. (ed.): Protecting Historic Architecture and Museum Collections from Natural Disasters. Elsevier, Burlington (2014)
Bevan, R.: 10 Heritage Sites Lost to Disaster and War. Google Arts & Culture. https://artsandculture.google.com/theme/10-heritage-sites-lost-to-disaster-and-war/kALyuo79hhrkLQ?hl=en. Last accessed on 14 May 2020
Guillemot, C., Le Meur, O.: Image inpainting: overview and recent advances. IEEE Signal Process. Mag. 31(1), 127–144 (2013)
Chang, L., Chongxiu, Y.: New interpolation algorithm for image inpainting. Phys. Procedia. 22, 107–111 (2011)
Amanatiadis, A., Andreadis, I.: A survey on evaluation methods for image interpolation. Meas. Sci. Technol. 20(10), 104015 (2009)
Thévenaz, P., Blu, T., Unser, M.: Interpolation revisited [medical images application]. IEEE Trans. Med. Imaging. 19(7), 739–758 (2000)
Keys, R.: Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process. 29(6), 1153–1160 (1981)
Unser, M., Aldroubi, A., Eden, M.: Fast B-spline transforms for continuous image representation and interpolation. IEEE Trans. Pattern Anal. Mach. Intell. 3, 277–285 (1991)
Carey, W.K., Chuang, D.B., Hemami, S.S.: Regularity-preserving image interpolation. IEEE Trans. Image Process. 8(9), 1293–1297 (1999)
Zhang, L., Wu, X.: An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15(8), 2226–2238 (2006)
Zhang, X., Wu, X.: Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Trans. Image Process. 17(6), 887–896 (2008)
Hwang, J.W., Lee, H.S.: Adaptive image interpolation based on local gradient features. IEEE Signal Process. Lett. 11(3), 359–362 (2004)
Dong, W., Zhang, L., Lukac, R., Shi, G.: Sparse representation-based image interpolation with nonlocal autoregressive modeling. IEEE Trans. Image Process. 22(4), 1382–1394 (2013)
Dyn, N., Levin, D., Rippa, S.: Data dependent triangulations for piecewise linear interpolation. IMA J. Numer. Anal. 10(1), 137–154 (1990)
Takagi, H., Aoyama, S., Makino, R., Hatsuda, T., Nakata, S., Tanaka, S.: Field approximation using piecewise polynomials for fast volume rendering on GPU. In: Advanced Methods, Techniques, and Applications in Modeling and Simulation, pp. 498–505. Springer, Tokyo (2012)
Akima, H.: A new method of interpolation and smooth curve fitting based on local procedures. J. ACM (JACM). 17(4), 589–602 (1970)
Fritsch, F.N., Carlson, R.E.: Monotone piecewise cubic interpolation. SIAM J. Numer. Anal. 17(2), 238–246 (1980)
Amidror, I.: Scattered data interpolation methods for electronic imaging systems: a survey. J. Electron. Imaging. 11(ARTICLE), 157–176 (2002)
Janarthanan, V., Jananii, G.: A detailed survey on various image inpainting techniques. Bonfring Int. J. Adv. Image Process. 2(2), 01–03 (2012)
Schönlieb, C.B.: Partial Differential Equation Methods for Image Inpainting, vol. 29. Cambridge University Press, Cambridge (2015)
Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, p. I (2001)
Caselles, V., Morel, J.M., Sbert, C.: An axiomatic approach to image interpolation. IEEE Trans. Image Process. 7(3), 376–386 (1998)
Masnou, S., Morel, J.M.: Level lines based disocclusion. In: Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No. 98CB36269), pp. 259–263 (1998)
Shen, J., Chan, T.F.: Mathematical models for local nontexture inpaintings. SIAM J. Appl. Math. 62(3), 1019–1043 (2002)
Chan, T.F., Shen, J.: Nontexture inpainting by curvature-driven diffusions. J. Vis. Commun. Image Represent. 12(4), 436–449 (2001)
Fadili, M.J., Starck, J.L., Murtagh, F.: Inpainting and zooming using sparse representations. Comput. J. 52(1), 64–79 (2009)
Shen, B., Hu, W., Zhang, Y., Zhang, Y.J.: Image inpainting via sparse representation. In: 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 697–700 (2009)
Fadili, M.J., Starck, J.L.: Em algorithm for sparse representation-based image inpainting. In: IEEE International Conference on Image Processing 2005, vol. 2, pp. II–61 (2005)
Ogawa, T., Haseyama, M.: Image inpainting based on sparse representations with a perceptual metric. EURASIP J. Adv. Signal Process. 2013(1), 179 (2013)
Zhang, J., Zhao, D., Gao, W.: Group-based sparse representation for image restoration. IEEE Trans. Image Process. 23(8), 3336–3351 (2014)
Mairal, J., Elad, M., Sapiro, G.: Sparse representation for color image restoration. IEEE Trans. Image Process. 17(1), 53–69 (2007)
Xu, Z., Sun, J.: Image inpainting by patch propagation using patch sparsity. IEEE Trans. Image Process. 19(5), 1153–1165 (2010)
Mahajan, K.S., Vaidya, M.B.: Image in painting techniques: a survey. IOSR J. Comput. Eng. (IOSRJCE). 5(4), 45–49 (2012)
Heeger, D.J., Bergen, J.R.: Pyramid-based texture analysis/synthesis. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 229–238 (1995)
Grossauer, H.: A combined PDE and texture synthesis approach to inpainting. In: European Conference on Computer Vision, pp. 214–224. Springer, Berlin/Heidelberg (2004)
Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1033–1038 (1999, September)
Ashikhmin, M.: Synthesizing natural textures. In: Proceedings of the 2001 Symposium on Interactive 3D Graphics, pp. 217–226 (2001)
Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 341–346 (2001)
Wei, L.Y., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 479–488 (2000)
Liang, L., Liu, C., Xu, Y.Q., Guo, B., Shum, H.Y.: Real-time texture synthesis by patch-based sampling. ACM Trans. Graph. (ToG). 20(3), 127–150 (2001)
Nealen, A., Alexa, M.: Hybrid Texture Synthesis, pp. 97–105. Techn. Univ., Fachbereich Informatik, Fachgebiet Graphisch-Interaktive Systeme (2003)
Yamauchi, H., Haber, J., Seidel, H.P.: Image restoration using multiresolution texture synthesis and image inpainting. In: Proceedings Computer Graphics International 2003, pp. 120–125 (2003, July)
Gatys, L., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 262–270. Morgan Kaufmann Publishers, San Mateo (2015)
Jetchev, N., Bergmann, U., Vollgraf, R.: Texture synthesis with spatial generative adversarial networks. arXiv preprint arXiv: 1611.08207 (2016)
Laube, P.: CNN texture synthesis for high-resolution image inpainting. In: Machine Learning Methods for Reverse Engineering of Defective Structured Surfaces, pp. 121–141. Springer Vieweg, Wiesbaden (2020)
Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)
Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings, vol. 2, p. II (2003)
Cheng, W.H., Hsieh, C.W., Lin, S.K., Wang, C.W., Wu, J.L.: Robust algorithm for exemplar-based image inpainting. In: Proceedings of International Conference on Computer Graphics, Imaging and Visualization, pp. 64–69 (2005)
Chen, Q., Zhang, Y., Liu, Y.: Image inpainting with improved exemplar-based approach. In: International Workshop on Multimedia Content Analysis and Mining, pp. 242–251. Springer, Berlin/Heidelberg (2007)
Hung, J.C., Huang, C.H., Liao, Y.C., Tang, N.C., Chen, T.J.: Exemplar-based image inpainting base on structure construction. JSW. 3(8), 57–64 (2008)
Wong, A., Orchard, J.: A nonlocal-means approach to exemplar-based inpainting. In: 2008 15th IEEE International Conference on Image Processing, pp. 2600–2603 (2008)
Goyal, P., Diwakar, S.: Fast and enhanced algorithm for exemplar-based image inpainting. In: 2010 Fourth Pacific-Rim Symposium on Image and Video Technology, pp. 325–330 (2010)
Yin, L., Chang, C.: An effective exemplar-based image inpainting method. In: 2012 IEEE 14th International Conference on Communication Technology, pp. 739–743 (2012, November)
Liu, Y., Caselles, V.: Exemplar-based image inpainting using multiscale graph cuts. IEEE Trans. Image Process. 22(5), 1699–1711 (2012)
Zhang, Q., Lin, J.: Exemplar-based image inpainting using color distribution analysis. J. Inf. Sci. Eng. 28(4), 641–654 (2012)
Choi, J.H., Hahm, C.H.: An exemplar-based image inpainting method with search region prior. In: 2013 IEEE 2nd Global Conference on Consumer Electronics (GCCE), pp. 68–71 (2013)
Wang, J., Lu, K., Pan, D., He, N., Bao, B.K.: Robust object removal with an exemplar-based image inpainting approach. Neurocomputing. 123, 150–155 (2014)
Patel, A.G., Kumar, S., Prajapati, A.D.: Improved exemplar-based image inpainting using structure tensor. Int. J. Comput. Appl. 96(15), 9–14 (2014)
Patel, J., Sarode, T.K.: Exemplar based image inpainting with reduced search region. Int. J. Comput. Appl. 92(12), 27–33 (2014)
Liang, Z., Yang, G., Ding, X., Li, L.: An efficient forgery detection algorithm for object removal by exemplar-based image inpainting. J. Vis. Commun. Image Represent. 30, 75–85 (2015)
Zhang, D., Liang, Z., Yang, G., Li, Q., Li, L., Sun, X.: A robust forgery detection algorithm for object removal by exemplar-based image inpainting. Multimed. Tools Appl. 77(10), 11823–11842 (2018)
Ahmed, M.W., Abdulla, A.A.: Quality improvement for exemplar-based image inpainting using a modified searching mechanism. UHD J. Sci. Technol. 4(1), 1–8 (2020)
Buyssens, P., Daisy, M., Tschumperlé, D., Lézoray, O.: Exemplar-based inpainting: technical review and new heuristics for better geometric reconstructions. IEEE Trans. Image Process. 24(6), 1809–1824 (2015)
Shroff, M., Bombaywala, M.S.R.: A qualitative study of exemplar based image inpainting. SN Appl. Sci. 1(12), 1730 (2019)
Chhabra, J.K., Birchha, M.V.: Detailed survey on exemplar-based image inpainting techniques. Int. J. Comput. Sci. Inf. Technol. 5(5), 6350–6635 (2014)
Jia, J., Tang, C.K.: Inference of segmented color and texture description by tensor voting. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 771–786 (2004)
Wu, J., Ruan, Q.: A novel hybrid image inpainting model. In: 2008 International Conference on Audio, Language and Image Processing, pp. 138–142 (2008)
Devasruthi, D., Menon, H.P., Narayanankutty, K.A.: FE-BEMD and exemplar-based hybrid image inpainting for occlusion removal. Int. J. Comput. Appl. 28(8), 38–44 (2011)
Zhao, M., Li, S.: Hybrid inpainting algorithm based on sparse representation and fast inpainting method. Int. J. Dig. Content Technol. Appl. 5(7), 239–247 (2011)
Sun, J., Yuan, L., Jia, J., Shum, H.Y.: Image completion with structure propagation. In: ACM SIGGRAPH 2005 Papers, pp. 861–868 (2005)
Richard, M.M.O.B.B., Chang, M.Y.S.: Fast digital image inpainting. In: Appeared in the Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain, pp. 106–107 (2001)
Bornemann, F., März, T.: Fast image inpainting based on coherence transport. J. Math. Imaging Vision. 28(3), 259–278 (2007)
Patel, H.N.: A survey on different techniques for image inpainting. Int. Res. J. Eng. Technol. (IRJET). 3, 340 (2016)
Cai, N., Su, Z., Lin, Z., Wang, H., Yang, Z., Ling, B.W.K.: Blind inpainting using the fully convolutional neural network. Vis. Comput. 33(2), 249–261 (2017)
Sasaki, K., Iizuka, S., Simo-Serra, E., Ishikawa, H.: Joint gap detection and inpainting of line drawings. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5725–5733 (2017)
Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2802–2810. Morgan Kaufmann Publishers, San Mateo (2016)
Sidorov, O., Yngve Hardeberg, J.: Deep hyperspectral prior: single-image denoising, inpainting, super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0 (2019)
Zhu, X., Qian, Y., Zhao, X., Sun, B., Sun, Y.: A deep learning approach to patch-based image inpainting forensics. Signal Process. Image Commun. 67, 90–99 (2018)
Ren, J.S., Xu, L., Yan, Q., Sun, W.: Shepard convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 901–909 (2015)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680. Morgan Kaufmann Publishers, San Mateo (2014)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge, MA (2016)
Jboor, N.H., Belhi, A., Al-Ali, A.K., Bouras, A., Jaoua, A.: Towards an inpainting framework for visual cultural heritage. In: 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), pp. 602–607 (2019)
Liu, H., Lu, G., Bi, X., Yan, J., Wang, W.: Image inpainting based on generative adversarial networks. In: 2018 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 373–378 (2018)
Demir, U., Unal, G.: Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422 (2018)
Parisotto, S.: MATLAB Codes for the Image Inpainting Problem. https://www.mathworks.com/matlabcentral/fileexchange/55326-matlab-codes-for-the-image-inpainting-problem (2020). MATLAB Central File Exchange. Retrieved May 18, 2020
Schönlieb, C.-B. Higher-Order Total Variation Inpainting. https://www.mathworks.com/matlabcentral/fileexchange/34356-higher-order-total-variation-inpainting (2020). MATLAB Central File Exchange. Retrieved May 18, 2020
Basak, S.: ImageCompletion-DCGAN. Github. https://github.com/saikatbsk/ImageCompletion-DCGAN (2017). Accessed 15 May 2020
Nicolas, B.: Image Inpainting via Dictionary Learning and Sparse Representation. Github. https://github.com/NicolasBizzozzero/Inpainting (2019)
zyh, Awesome-Inpainting-Tech. Github. https://github.com/1900zyh/Awesome-Image-Inpainting (2020)
Huang, C., Yoshida, K.: Evaluations of Image Completion Algorithms: Exemplar-Based Inpainting vs. Deep Convolutional GAN
Han, D.: Comparison of commonly used image interpolation methods. In: Proceedings of the 2nd International Conference on Computer Science and Electronics Engineering. Atlantis Press (2013)
Cuomo, S., Galletti, A., Giunta, G., Marcellino, L.: A novel triangle-based method for scattered data interpolation. Appl. Math. Sci. 8(134), 6717–6724 (2014)
WikiArt.org: WikiArt.org – visual art encyclopedia. Available: https://www.wikiart.org/ (15 May 2020)
Pixnio: Free images for anyone and any use. Available: https://pixnio.com/ (15 May 2020)
Pxfuel: Royalty free stock photos free & unlimited download. Available: https://www.pxfuel.com/ (15 May 2020)
Wang, Z., et al.: Image quality assessment: from error visibility to structural similarity. In: Image Processing, IEEE Transactions on 13.4, pp. 600–612 (2004)
Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369 (2010)
Acknowledgement
This work was made possible by NPRP grant 9-181-1-036 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors (www.ceproqha.qa).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Ahmed, H.O., Alfaqheri, T., Sadka, A.H. (2021). Digital Image Inpainting Techniques for Cultural Heritage Preservation and Restoration. In: Belhi, A., Bouras, A., Al-Ali, A.K., Sadka, A.H. (eds) Data Analytics for Cultural Heritage. Springer, Cham. https://doi.org/10.1007/978-3-030-66777-1_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-66777-1_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66776-4
Online ISBN: 978-3-030-66777-1
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)