Skip to main content
SpringerLink
Log in

Auto-inpainting heritage scenes: a complete framework for detecting and infilling cracks in images and videos with quantitative assessment

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

The need for preservation of cultural heritage has necessitated the research on digitally repairing the photographs of damaged monuments. In this paper, we first propose a technique for automatically detecting the cracked regions in photographs of monuments. Unlike the usual practice of manually selecting the mask for inpainting, the detected regions are supplied to an inpainting algorithm. Thus, the process of digitally repairing the cracked regions that physical objects have, using inpainting, is completely automated. The detection of cracked regions is based on comparison of patches, for which we use a measure derived from the edit distance, which is a popular string metric used in the area of text mining. Further, we extend this method to perform inpainting of video frames by making use of the scale-invariant feature transform and homography. We consider the camera to move while capturing video of the heritage site, as such videos are typically captured by novices, hobbyists and tourists. Finally, we also propose a video quality measure to quantify the temporal consistency of the inpainted video. Experiments have been carried out on videos captured from the heritage site at Hampi, India.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

Notes

  1. The Details of selecting a suitable tolerance value \({\delta _{t}}\) are given in Sect. 6.

  2. For active contour segmentation technique, we have used the implementation available at http://www.mathworks.in/matlabcentral/fileexchange/23847-sparse-field-methods-for-active-contours.

  3. An implementation for extraction and matching of SIFT keypoints and corresponding descriptor is available at http://www.cs.ubc.ca/~lowe/keypoints/.

  4. For fitting homography to keypoints using RANSAC, we used the code available at http://www.csse.uwa.edu.au/~pk/Research/MatlabFns/Robust/ransacfithomography.m.

  5. For decomposition of estimated homography, we have used the implementation available at http://cs.gmu.edu/~kosecka/examples-code/homography2Motion.m.

  6. The details of selecting threshold \({\delta _{r}}\) are given in Sect. 6.

References

  1. Amano, T.: Correlation based image defect detection. In: Proceedings of the 18th International Conference on Pattern Recognition. ICPR ’06, vol. 01, pp. 163–166. IEEE Computer Society, Washington, DC, USA (2006)

  2. Ammouche, A., Riss, J., Breysse, D., Marchand, J.: Image analysis for the automated study of microcracks in concrete. Cement Concrete Comp. 23(23), 267–278 (2001) (Special theme issue on image analysis)

  3. Bendels, G.H., Guthe, M., Klein, R.: Free-form modelling for surface inpainting. In: Proceedings of the 4th International Conference on Computer Graphics. Virtual reality, visualisation and interaction in Africa, AFRIGRAPH ’06, pp. 49–58. ACM, New York, NY, USA (2006)

  4. Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. SIGGRAPH ’00, pp. 417–424. ACM Press/Addison–Wesley Publishing Co., New York, NY, USA (2000)

  5. Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. Int. J. Comput. Vis. 74(1), 59–73 (2007)

    Article  Google Scholar 

  6. Chan, T., Vese, L.: Active contours without edges. Image Process. IEEE Trans. 10(2), 266–277 (2001)

    Article  MATH  Google Scholar 

  7. Chang, R.C., Sie, Y.L., Chou, S.M., Shih, T.K.: Photo defect detection for image inpainting. In: Proceedings of the Seventh IEEE International Symposium on Multimedia. ISM ’05, pp. 403–407. IEEE Computer Society, Washington, DC, USA (2005)

  8. Cornelis, B., Rui, T., Gezels, E., Dooms, A., Pi++urica, A., Platia, L., Cornelis, J., Martens, M., Mey, M.D., Daubechies, I.: Crack detection and inpainting for virtual restoration of paintings: the case of the ghent altarpiece. Sig. Process. 93(3), 605–619 (2013) (Image processing for digital art work)

  9. Criminisi, A., Pérez, P., Toyama, K.: Object removal by exemplar-based inpainting. Comput. Vis. Pattern Recognit. IEEE Comput. Soc. Conf. 2, 721 (2003)

    Google Scholar 

  10. Criminisi, A., Pérez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 1200–1212 (2004)

    Article  Google Scholar 

  11. El-Hakim, S.F.: Semi-automatic 3d reconstruction of occluded and unmarked surfaces from widely separated views. In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Greece, pp. 143–148 (2002)

  12. El-Hakim, S.F., MacDonald, G., Lapointe, J.F., Gonzo, L., Jemtrud, M.: On the digital reconstruction and interactive presentation of heritage sites through time. In: VAST 2006: the 7th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage, Nicosia, Cyprus, 2006. Proceedings, pp. 243–250. Eurographics Association (2006)

  13. Farias, M., Mitra, S.: No-reference video quality metric based on artifact measurements. In: Image Processing, 2005. ICIP 2005. IEEE International Conference on, vol. 3, pp. III-141–144 (2005)

  14. Faugeras, O., Lustman, F.: Motion and structure from motion in a piecewise planar environment. Tech. Rep. RR-0856, INRIA (1988)

  15. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  16. Grossauer, H.: A combined pde and texture synthesis approach to inpainting. Eur. Conf. Comput. Vis. 4, 214–224 (2004)

    Google Scholar 

  17. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, New York (2003)

    Google Scholar 

  18. Jeannin, S., Divakaran, A.: MPEG-7 visual motion descriptors. IEEE Trans. Circuits Syst. Video Technol. 11(6), 720–724 (2001). doi:10.1109/76.927428

    Article  Google Scholar 

  19. de Kadt, C., Gain, J., Marais, P.: Revisiting district six: a case study of digital heritage reconstruction from archival photographs. In: Proceedings of the 6th International Conference on Computer Graphics. Virtual reality, visualisation and interaction in Africa, AFRIGRAPH ’09, pp. 13–21. ACM, New York, NY, USA (2009)

  20. Kang, H.W., Shin, S.Y.: Creating walk-through images from a video sequence of a dynamic scene. Presence: telepresence, and automotive. Environments 13, 638–655 (2004)

    Google Scholar 

  21. Kenney, J.F.: Mathematicals of Statistics. Van Nostrand (1954)

  22. Kovesi, P.D.: MATLAB and octave functions for computer vision and image processing. Centre for Exploration Targeting, School of Earth and Environment, The University of Western Australia (2005). http://www.csse.uwa.edu.au/pk/research/matlabfns/robust/ransacfithomography

  23. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  24. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence, vol. 2, IJCAI’81, pp. 674–679 (1981)

  25. Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S.: An invitation to 3-D vision: from images to geometric model. Springer (2003)

  26. Masnou, S., Morel, J.M.: Level lines based disocclusion. Image processing, 1998. ICIP 98. In: Proceedings 1998 International Conference on pp. 259–263 (1998)

  27. McCloskey, S., Langer, M., Siddiqi, K.: Removal of partial occlusion from single images. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 647–654 (2011). doi:10.1109/TPAMI.2010.187

    Article  Google Scholar 

  28. Oliveira, M.M., Bowen, B., Mckenna, R., Sung Chang, Y.: Fast digital image inpainting. In: Proceedings of the International Conference on Visualization, Imaging and Image Processing VIIP 2001, pp. 261–266, ACTA Press (2001)

  29. Padalkar, M.G., Vora, M.V., Joshi, M.V., Zaveri, M.A., Raval, M.S.: Identifying vandalized regions in facial images of statues for inpainting. New Trends in Image Analysis and Processing-ICIAP 2013. Lecture Notes in Computer Science, vol. 8158, pp. 208–217, Springer, Berlin (2013)

  30. Padalkar, M.G., Zaveri, M.A., Joshi, M.V.: Svd based automatic detection of target regions for image inpainting. In: Park, J.I., Kim, J. (eds.) Computer vision - ACCV 2012 Workshops. Lecture Notes in Computer Science, vol. 7729, pp. 61–71. Springer, Berlin (2013)

  31. Parmar, C.M., Joshi, M.V., Raval, M.S., Zaveri, M.A.: Automatic image inpainting for the facial images of monuments. In: Proceedings of Electrical Engineering Centenary Conference 2011, pp. 415–420. IISc Bangalore, India (2011)

  32. Patwardhan, K.A., Sapiro, G., Bertalmío, M.: Video inpainting under constrained camera motion. IEEE Trans. Image Process. 16(2), 545–553 (2007)

    Article  MathSciNet  Google Scholar 

  33. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)

    Article  Google Scholar 

  34. Ringot, E., Bascoul, A.: About the analysis of microcracking in concrete. Cement Concrete Comp. 23(23), 261–266 (2001) (Special theme issue on image analysis)

  35. Rother, C., Kolmogorov, V., Blake, A.: Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3), 309–314 (2004)

    Article  Google Scholar 

  36. Saad, M., Bovik, A., Charrier, C.: Blind prediction of natural video quality. Image Process. IEEE Trans. 23(3), 1352–1365 (2014). doi:10.1109/TIP.2014.2299154

    Article  MathSciNet  Google Scholar 

  37. Saad, M.A., Bovik, A.C.: Blind quality assessment of videos using a model of natural scene statistics and motion coherency. In: Asilomar Conference on Signals, Systems, and Computers, pp. 332–336 (2012)

  38. Scholes, S.: Mcconkie ranch petroglyphs near vernal, utah (2011). https://www.youtube.com/watch?v=jmewuqEXTK8. Accessed 01 Sept 2014

  39. Shibata, T., Iketani, A., Senda, S.: Image in painting based on probabilistic structure estimation. In: Proceedings of the 10th Asian conference on Computer vision, vol. III. ACCV’10, pp. 109–120. Springer, Berlin (2011)

  40. Shih, T.K., Tang, N.C., Yeh, W.S., Chen, T.J., Lee, W.: Video inpainting and implant via diversified temporal continuations. In: Proceedings of the 14th Annual ACM International Conference on Multimedia. MULTIMEDIA ’06, pp. 133–136. ACM, New York, NY, USA (2006)

  41. Sikora, T.: MPEG digital video-coding standards. Sig. Process. Mag. IEEE 14(5), 82–100 (1997)

    Article  Google Scholar 

  42. Tamaki, T., Suzuki, H., Yamamoto, M.: String-like occluding region extraction for background restoration. Int. Conf. Pattern Recognit. 3, 615–618 (2006)

    Google Scholar 

  43. Turakhia, N., Shah, R., Joshi, M.: Automatic crack detection in heritage site images for image inpainting. In: Eighth Indian Conference on Computer Vision, Graphics and Image Processing (ICVGIP), p. 68 (2012)

  44. Wagner, R.A., Fischer, M.J.: The string-to-string correction problem. J. ACM 21(1), 168–173 (1974)

    Article  MATH  MathSciNet  Google Scholar 

  45. Wang, Z., Zhou, F., Qi, F.: Inpainting thick image regions using isophote propagation. In: Image Processing 2006 IEEE International Conference on, pp. 689–692 (2006). doi:10.1109/ICIP.2006.312428

  46. Wexler, Y., Shechtman, E., Irani, M.: Space-time completion of video. Pattern Anal. Mach. Intell. IEEE Trans. 29(3), 463–476 (2007)

    Article  Google Scholar 

  47. Wu, J., Ruan, Q.: Object removal by cross isophotes exemplar-based inpainting. In: Proceedings of the 18th International Conference on Pattern Recognition. ICPR ’06, pp. 810–813. IEEE Computer Society, Washington, DC, USA (2006)

  48. Zou, Q., Cao, Y., Li, Q., Mao, Q., Wang, S.: Cracktree: Automatic crack detection from pavement images. Pattern Recogn. Lett. 33(3), 227–238 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dhirubhai Ambani Institute of Information and Communication Technology, Gandhinagar, Gujarat, India

    Milind G. Padalkar & Manjunath V. Joshi

Authors
  1. Milind G. Padalkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manjunath V. Joshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manjunath V. Joshi.

Additional information

This work is a part of the project Indian Digital Heritage (IDH)-Hampi sponsored by Department of Science and Technology (DST), Govt. of India (Grant No: NRDMS/11/1586/2009/Phase-II).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Padalkar, M.G., Joshi, M.V. Auto-inpainting heritage scenes: a complete framework for detecting and infilling cracks in images and videos with quantitative assessment. Machine Vision and Applications 26, 317–337 (2015). https://doi.org/10.1007/s00138-015-0661-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-015-0661-6

Keywords

Search

Navigation