Acquisition, Representation, Processing and Display of Digital Heritage Sites

Chapter

Abstract

This chapter presents salient aspects of the research undertaken on the project Acquisition, Representation, Processing and Display of Digital Heritage Sites. The main objective of the project was to create algorithms and techniques to acquire a three-dimensional digital replica of complex structures spread over a large area. The techniques developed are applied to Hampi, a world heritage site. In addition to acquiring the geometry and surface properties, we also research efficient representation and visualisation of this data and provide tools and methods for users to experience the captured models, to virtual walk-through and explore the digital recreations. For the acquisition, we rely on multimodal input using technologies like laser scanners, colour cameras and depth sensors. We align and fuse geometric constructions from different modalities through a step of registration. We have extended structure from motion (SfM), a state-of-the-art approach for multi-view 3D reconstruction from images and developed techniques for large-scale (relatively) sparse geometric constructions and simultaneously dense reconstructions of smaller parts. We also provide ability to generate high-resolution point cloud from the point cloud obtained from depth camera Kinect by using additional high definition cameras. We also explore efficient visualisation of large models with augmented reality and user experience authoring. Hampi has been chosen as a test bench for developing our techniques. Within Hampi, we concentrate on Vittala Temple Complex, and demonstrate our techniques on it. The project has greatly benefitted from the collaboration from other partner institutes especially BVBCET, NID, NIAS, IIT Bombay, IIT Madras and IIIT Hyderabad and IISc Bangalore.

Notes

Acknowledgements

The sponsorship and continuous support from Department of Science and Technology for the project are highly appreciated. The project has greatly benefitted from the collaboration of other institutes. In particular, BVBCET (Prof. Uma Mudenagudi), NIAS (Prof. Meera Natampally), IIIT Hyderabad (Prof. Anoop Namboodiri), IIT Bombay (Prof. Parag Chaudhuri), IISc Bangalore (Prof. Venu Madhav Govindu). The implementation and development required efforts of many research scholars and students. These efforts are parts of several Ph.D. and Masters theses. These include Brojeshwar Bhowmick (Ph.D.), Suvam Patra (Ph.D.), Nishant Bugalia (MSR 2016), Shantanu Chaudhari (M.Tech. 2016), Abhinav Shukla (M.Tech. 2011), Harsh Vardhan (M.Tech. 2011), Lissy Verma (M.Tech. 2011), Rahul Kumar (M.Tech. 2011), Nidhi Arora (M.Tech. 2011), Ankush Kumar (M.Tech. 2011), Anay Ghotikar (M.Tech. 2012), Ankit (M.Tech. 2012), Suvam Patra (M.Tech. 2012), Neeraj Kulkarni (M.Tech. 2012), Shruti Agarwal (M.Tech. 2012), Richa Gupta (M.Tech. 2013), Ramji Gupta (M.Tech. 2013), Kinshuk Sarabhai (MSR 2013) and Satyendra Singh (M.Tech. 2014).

References

  1. 1.
    Agarwal S, Snavely N, Seitz S, Szeliski R (2010) Bundle adjustment in the large. In: Proceedings of the European conference on computer vision, pp 29–42Google Scholar
  2. 2.
    Agarwal S, Snavely N, Simon I, Seitz S, Szeliski R (2009) Building rome in a day. In: Proceedings of the international conference on computer vision, pp 72–79Google Scholar
  3. 3.
    Arora N, Kalra P (2011) Interactive image restoration using inpainting and denoising. In: NCVPRIPG 2011—Proceedings of national conference on computer vision, pattern recognition, image processing and graphics, Hubli, 15–17 December, 2011, pp 219–222Google Scholar
  4. 4.
    Arora N, Kumar A, Kalra P (2013) Digital restoration of old paintings. In: WSCG 2013—Proceedings of conference on computer graphics, visualization and computer vision plzen, Czech Republic 24–27 June, 2013, p F89Google Scholar
  5. 5.
    Bhowmick B, Patra S, Chaterjee A, Govindu VM, Banerjee S (2014) Divide and conquer: efficient large scalre structure from motion using graph partitioning. Proc ACCV 2014:273–287Google Scholar
  6. 6.
    Boykov Y, Kolmogorov V (2004) An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In: IEEE transactions on PAMI, vol 26, no 9, September 2004, pp 1124–1137Google Scholar
  7. 7.
    Bugalia N (2016) Immersive environment system for an efficient human computer interaction. MSR thesis, Amar Nath and Shashi Khosla School of Information Technology, IIT DelhiGoogle Scholar
  8. 8.
    Bugalia N, Kumar S, Kalra P, Choudhary S (2016) Mixed reality based interaction system for digital heritage. In: Proceedings of the 15th ACM SIGGRAPH conference on virtual-reality continuum and its applications in industry, VRCAI 2016, Zhuhai, China, December 3–4, 2016, pp 31–37Google Scholar
  9. 9.
    Crandall DJ, Owens A, Snavely N, Huttenlocher DP (2011) Discrete-continuous optimization for large-scale structure from motion. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3001–3008Google Scholar
  10. 10.
    Crandall DJ, Owens A, Snavely N, Huttenlocher DP (2013) SfM with MRFs: discrete-continuous optimization for large-scale reconstruction. IEEE Trans Pattern Anal Mach Intell 35(12):2841–2853CrossRefGoogle Scholar
  11. 11.
    Criminisi A, Reid I, Zisserman A (2000) Single view metrology. Int J Comput Vis 40(2):123–148Google Scholar
  12. 12.
    Diebel J, Thrun S (2006) An application of markov random fields to range sensing. In: Advances in neural information processing systems, pp 291–298Google Scholar
  13. 13.
    Farenzena M, Fusiello A, Gherardi R (2009) Structure-and-motion pipeline on a hierarchical cluster tree. In: Proceedings of IEEE international conference on computer vision workshop on 3-D digital imaging and modeling, pp 1489–1496Google Scholar
  14. 14.
    Frahm J, Georgel P, Gallup D, Johnson T, Raguram R, Wu C, Jen Y, Dunn E, Clipp B, Lazebnik S, Pollefeys M (2010) Building Rome on a cloudless day. In: Proceedings of the European conference on computer vision: Part IV, pp 368–381Google Scholar
  15. 15.
    Furukawa Y (2016) Clustering views for multi-view stereo. http://www.di.ens.fr/cmvs. Accessed 26 July 26 2016
  16. 16.
    Havlena M, Torii A, Pajdla T (2010) Efficient structure from motion by graph optimization. In: Proceedings of the European conference on computer vision, Lecture notes in computer science, vol 6312, pp 100–113Google Scholar
  17. 17.
    Hoiem D, Efros AA, Hebert M (2005) Automatic photo pop-up. Proc SIGGRAPH 2005:577–584Google Scholar
  18. 18.
    Horry Y, Anjyo KI, Arai K (1997) Tour into the picture: using a spidery mesh interface to make animation from a single image. Proc SIGGRAPH 97:225–232Google Scholar
  19. 19.
    Jiang N, Cui Z, Tan P (2013) A global linear method for camera pose registration. In: Proceedings of IEEE international conference on computer vision, pp 481–488Google Scholar
  20. 20.
    Khurana D, Sankhla S, Shukla A, Varshney R, Kalra P, Banerjee S (2012) A grammar-based gui for single view reconstruction. In: Proceedings of ICVGIP 2012, p 14Google Scholar
  21. 21.
    Kushal A, Chanda G, Srivastava K, Gupta M, Sanyal S, Sriram T, Kalra P, Banerjee S (2003) Multilevel modelling and rendering of architectural scenes. In: Proceedings of EUROGRAPHICS 2003 short presentationsGoogle Scholar
  22. 22.
    Kutulakos K, Seitz S (1999) A theory of shape by space carving. In: 7th IEEE international conference on computer vision (ICCV-99), vol I, pp 307–314Google Scholar
  23. 23.
    Laycock RG, Drinkwater D, Day AM (2008) Exploring cultural heritage sites through space and time. J Comput Cult Heritage 1(2):1–15CrossRefGoogle Scholar
  24. 24.
    Lourakis M, Argyros A (2007) Enforcing scene constraints in single view reconstruction. In: Proceedings of EUROGRAPHICS 2007 short papers, pp 45–48Google Scholar
  25. 25.
    Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRefGoogle Scholar
  26. 26.
    Magnenat-Thalmann N, Foni AE, Cadi-Yazli N (2006) Real-time animation of ancient roman sites. In: Proceedings of 4th international conference on computer graphics and interactive techniques, GRAPHITE, pp 19–30Google Scholar
  27. 27.
    Moulon P, Monasse P, Marlet R (2013) Global fusion of relative motions for robust, accurate and scalable structure from motion. In: Proceedings of IEEE international conference on computer vision, pp 3248–3255Google Scholar
  28. 28.
    Muller P, Zeng G, Wonka P, Gool LV (2007) Image-based procedural modeling of facades. Proc SIGGRAPH 2007:181–184Google Scholar
  29. 29.
    Nister D, Stewenius H (2006) Scalable recognition with a vocabulary tree. Proc IEEE Conf Comput Vis Pattern Recogn 2:2161–2168Google Scholar
  30. 30.
    Papagiannakis G, Schertenleib S, O’Kennedy B, Arevalo-Pozat M, Magnenat-Thalmann N, Thalmann D (2005) Mixing virtual and real scenes in the site of ancient pompeii. Comput Anim Virtual Worlds 16(1):11–24CrossRefGoogle Scholar
  31. 31.
    Patra S, Bhowmick B, Kalra P, Banerjee S (2012) Kinect. In: VISIAPP 2012Google Scholar
  32. 32.
    Petridis P, White M, Mourkousis N, Liarokapis F, Sifiniotis M, Gatzidis C (2009) Exploring and interacting with virtual museums. J Comput Cult Heritage 2(1):1–20Google Scholar
  33. 33.
    Raghuram R, Wu C, Frahm J, Lazebnik S (2011) Modeling and recognition of landmark image collections using iconic scene graphs. Int J Comput Vis 95(3):213–239CrossRefGoogle Scholar
  34. 34.
    Schuon S, Theobalt C, Davis J, Thrun S (2008) High-quality scanning using time-of-flight depth superresolution. In: IEEE Computer society conference on computer vision and pattern recognition workshops 2008Google Scholar
  35. 35.
    Schuon S, Theobalt C, Davis J, Thrun S (2009) Lidarboost depth superresolution for tof 3d shape scanning. In: CVPR 2009Google Scholar
  36. 36.
    Sharma A, Kumar S (2014) User-guided modulation of rendering techniques for detail inspection. In: GRAPP 2014—Proceedings of the 9th international conference on computer graphics theory and applications, Lisbon, Portugal, 5–8 January, 2014, pp 247–254Google Scholar
  37. 37.
    Shi J, Malik J (2000) Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intell 22(8):888–905CrossRefGoogle Scholar
  38. 38.
    Slabaugh G, Schafer R (2004) Methods for volumetric reconstruction of visual scenes. Int J Comput Vis 179–199Google Scholar
  39. 39.
    Smisek J, Jancosek M, Pajdla T (2011) 3d with kinect. In: IEEE workshop on consumer depth cameras for computer visionGoogle Scholar
  40. 40.
    Snavely N, Seitz S, Szeliski R (2006) Photo tourism: exploring photo collections in 3d. In: Proceedings of ACM SIGGRAPH, pp 835–846Google Scholar
  41. 41.
    Snavely N, Seitz S, Szeliski R (2008) Modeling the world from internet photo collections. Int J Comput Vis 80(2):189–210CrossRefGoogle Scholar
  42. 42.
    Snavely N, Seitz S, Szeliski R (2008) Skeletal graphs for efficient structure from motion. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1–8Google Scholar
  43. 43.
    Sturm PF, Maybank SJ (1999) A method for interactive 3d reconstruction of piecewise planar objects from single images. Proc BMVC 1999:265–274Google Scholar
  44. 44.
    Trapp M, Semmo A, Pokorski R, Hermann CD, Dollner J, Eichhorn M, Heinzelmann M (2010) Communication of digtial cutltural heritage in public spaces by the example of roman cologne. In: Proceedings of Euro Mediterranean conference, pp 262–276Google Scholar
  45. 45.
    Triggs B, Mclauchlan P, Hartley R, Fitzgibbon A (2000) Bundle adjustment a modern synthesis. In: Vision algorithms: theory and practice, LNCS, pp 298–372Google Scholar
  46. 46.
    Wu C (2007) SiftGPU: a GPU implementation of scale invariant feature transform (SIFT). http://cs.unc.edu/~ccwu/siftgpu
  47. 47.
    Wu C (2013) Towards linear-time incremental structure from motion. In: Proceedings of the international conference on 3D vision, 3DV ’13, pp 127–134Google Scholar
  48. 48.
    Wu C, Agarwal S, Curless B, Seitz S (2011) Multicore bundle adjustment. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3057–3064Google Scholar
  49. 49.
    Yang Q, Yang R, Davis J, Nistr D (2007) Spatial-depth super resolution for range images. In: CVPR 2007Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2017

Authors and Affiliations

  1. 1.Indian Institute of Technology DelhiNew DelhiIndia

Personalised recommendations