Reality-based 3D documentation of natural and cultural heritage sites—techniques, problems, and examples

The continuous development of new sensors, data capture methodologies, multi-resolution 3D representations, and the improvement of existing ones are contributing significantly to the documentation, conservation, and presentation of heritage information and to the growth of research in the cultural heritage field. This is also driven by the increasing requests and needs for digital documentation of archaeological sites at different scales and resolutions.

A technique is intended as a scientific procedure (e.g., image processing) to accomplish a specific task while a methodology is a group or combination of techniques and activities combined to achieve a particular task. Reality-based techniques (e.g., photogrammetry, laser scanning, etc.; Gruen 2008) employ hardware and software to survey the reality as it is, documenting the actual or as-built situation of a site and reconstructing it from real data. Non-real approaches are instead based on computer graphics software (3D Studio, Maya, Sketchup, etc.) or procedural modeling approaches (Mueller et al. 2006), and they allow the generation of 3D data without any particular survey or knowledge of a site (Fig. 2).

Open image in new window

Image-based methods

Image data require a mathematical formulation to transform the 2D image measurements into 3D information. Generally, at least two images (Fig. 3) are required, and 3D data can be derived using perspective or projective geometry formulations. Image-based modeling techniques (mainly photogrammetry and computer vision; Remondino and El-Hakim 2006) are generally preferred in cases of lost objects, monuments or architectures with regular geometric shapes, small objects with free-form shape, mapping applications, deformation analyses, low budgets, good experience of the working team, and time or location constraints for the data acquisition and processing. Between the available image acquisition platforms (space, airborne, and terrestrial), of particular interest are the UAVs, e.g., unmanned aerial vehicles (like model helicopters) which can fly in an autonomous mode, using integrated GPS/INS, stabilizer platform, and digital cameras (or even a small range sensor) and which can be used to get data from otherwise hardly accessible areas. Image-based 3D modeling generally requires some user’s interaction in the different steps of the 3D reconstruction and modeling pipeline, reducing its use mainly to experts. The pipeline (Fig. 4) is generally composed of different steps which can be performed in an automated or interactive way, according to the user requirements and project specifications. Accurate feature extraction from satellite and aerial images is still a manually driven procedure. In terrestrial applications, more automation is available for scene reconstruction. Fully automated methods based on a “structure from motion” approach (Pollefeys et al. 2004; Vergauwen and Van Gool 2006; Goesele et al. 2007; Agarwal et al. 2009) are getting quite common in the 3D heritage community, although mainly useful for visualization, object-based navigation, annotation transfer, or image browsing purposes and not for metric and accurate 3D reconstructions and documentations. However, the automation of the procedures has reached a significant maturity with the capability to orient huge numbers of images (Snavely et al. 2008), and open source programs are also available (e.g., Blunder or its graphical implementation Photosynth). But the complete automation in image-based modeling is still an open research’s topic, in particular for the 3D surveying and modeling of architectural scenes and man-made objects (Patias et al. 2008). Nevertheless, the camera calibration and image orientation steps can be achieved fully automatically (Barazzetti et al. 2009) as well as the surface measurement and the texturing for a large number of free-form objects (Remondino et al. 2008), but the user interaction is still necessary in the geo-referencing and for the quality control part.

Open image in new window

Open image in new window

3D information could also be derived from a single image using object constraints (Van den Heuvel 1998; Criminisi et al. 1999; El-Hakim 2000) or estimating surface normals instead image correspondences (shape from shading (Horn and Brooks 1989), shape from texture (Kender 1978), shape from specularity (Healey and Binford 1987), shape from contour (Meyers et al. 1992), and shape from 2D edge gradients (Winkelbach and Wahl 2001)).

Photogrammetry is considered the primary technique for the processing of image data, being able to deliver at any scale of application accurate and detailed 3D information with estimates of precision and reliability of the unknown parameters from measured image correspondences (tie points). The correspondences can be extracted automatically or semi-automatically according to the object and project requirements. Photogrammetry is employed in different applications like mapping, 3D documentation, conservation, digital restoration, reverse engineering, monitoring, visualization, animation, urban planning, deformation analysis, etc. Photogrammetric 3D reconstructions are generally performed with interactive procedures for man-made objects or architectural structures where sparse point clouds and few geometric primitives are sufficient to describe the 3D geometry (Fig. 5). Automated image matching procedures are instead employed for free-form objects where dense point clouds are required to correctly describe all the object discontinuities and features (Fig. 6).

Open image in new window

Open image in new window

Many authors (Pomaska 2001; D’Ayala and Smars 2003; Heritage 2005) reported how the photogrammetric image-based approach allows surveys at different levels and in all possible combinations of object complexities, with high quality requirements, easy usage and manipulation of the final products, few time restrictions, good flexibility, and low costs. Different comparisons between photogrammetry and range sensors were also presented in the literature (Böhler 2005; Remondino et al. 2005; Grussenmeyer et al. 2008).

Multi-sensor and multi-source data integration

As previously mentioned, nowadays, the state-of-the-art approach for the 3D documentation and modeling of large and complex sites uses and integrates multiple sensors and technologies (photogrammetry, laser scanning, topographic surveying, etc.) to (a) exploit the intrinsic potentials and advantages of each technique, (b) compensate for the individual weaknesses of each method alone, (c) derive different geometric levels of detail (LOD) of the scene under investigation, and (d) achieve more accurate and complete geometric surveying for modeling, interpretation, representation, and digital conservation issues. 3D modeling based on multi-scale data and multi-sensors integration is indeed providing the best 3D results in terms of appearance and geometric detail. Each LOD is showing only the necessary information while each technique is used where best suited.

Since the 1990s, multiple data sources were integrated for industrial, military, and mobile mapping applications. Sensor and data fusion were then applied also in the cultural heritage domain, mainly at terrestrial level (Stumpfel et al. 2003; El-Hakim et al. 2004), although some projects mixed and integrated satellite, aerial, and ground information for a more complete and multi-resolution 3D survey (Gruen et al. 2005; Rönholm et al. 2007; Guidi et al. 2009a).

The multi-sensor and multi-resolution concept (Fig. 7) should be distinguished between (a) geometric modeling (3D shape acquisition, registration, and further processing) where multiple resolutions and sensors are seamlessly combined to model features with the most adequate sampling step and derive different geometric LOD of the scene under investigation and (b) appearance modeling (texturing, blending, simplification, and rendering) where photo-realistic representations are sought taking into consideration variations in lighting, surface specularity, seamless blending of the textures, user’s viewpoint, simplification, and LOD.

Open image in new window

Beside images acquired in the visible part of the light spectrum, it is often necessary to acquire extra information provided by other sensors working in different spectral bands (e.g., IR, UV) in order to study deeper the object. Thermal infrared information is useful to analyze historical buildings, their state of conservation, reveal padding, older layers, back structure of frescoes while near IR is used to study paintings, revealing pentimenti, and preparatory drawings. On the other hand, the UV radiations are very useful in heritage studies to identify different varnishes and over-paintings, in particular with induced visible fluorescence imaging systems (Pelagotti et al. 2006). All those multi-modal information need to be aligned and often overlapped to the geometric data for information fusion, multi- spectral analysis, or other diagnostic applications (Remondino et al. 2009b).

Many image-based modeling packages as well as range-based systems came out on the market in the last decades to allow the digital documentation and 3D modeling of objects or scenes. Many new users are approaching these methodologies, and those who are not really familiar with them need clear statements and information to know if a package or system satisfies certain requirements before investing. Therefore, technical standards for the 3D imaging field must be created, like those available for the traditional surveying or CMM. A part from standards, comparative data, and best practices are also needed, to show not only advantages but also limitations of systems and software. In these respects, the German VDI/VDE 2634 contains acceptance testing and monitoring procedures for evaluating the accuracy of close-range optical 3D vision systems (particularly for full-frame range cameras and single scan). The American Society for Testing and Materials with its E57 standards committee is trying to develop standards for 3D imaging systems for applications like surveying, preservation, construction, etc. The International Association for Pattern Recognition (IAPR) created the Technical Committee 19—Computer Vision for Cultural Heritage Applications—with the goal of promoting Computer Vision Applications in Cultural Heritage and their integration in all aspects of IAPR activities. TC19 aims at stimulating the development of components (both hardware and software) that can be used by researchers in cultural heritage like archaeologists, art historians, curators, and institutions like universities, museums, and research organizations.

As far as the presentation and visualization of the achieved 3D models concerns, the London Charter (http://www.londoncharter.org/) is seeking to define the basic objectives and principles for the use of 3D visualization methods in relation to intellectual integrity, reliability, transparency, documentation, standards, sustainability, and access of cultural heritage.

The Open Geospatial Consortium (OGC) developed the GML3, an extensible international standard for spatial data exchange. GML3 and other OGC standards (mainly the OpenGIS Web Feature Service Specification) provide a framework for exchanging simple and complex 3D models. Based on the GML3, the CityGML standard was created, an open data model and XML-based format for storing, exchanging, and representing 3D urban objects and in particular virtual city models.

In case of satellite and aerial images, the availability of the data could be a problem due to weather conditions or restrictions on flights. For terrestrial acquisition, size, location, and surface (geometry and material) of the object or site can create several problems. The dimensions and accessibility problems (due to location, obstructions, rough or sloped terrain with stones, rocks and holes, unfavorable weather conditions, etc.) can cause delays, occlusions, and can result in missing sections or enforce wide-baseline images and poor geometric configurations. The complexity of some parts can create self-occlusions or holes in the coverage, in addition to the occlusions from plants, trees, restoration scaffolds, or tourists. The absence of high platforms for a higher location of the data acquisition might cause missing parts, e.g., for the roofs.

For active sensors, the object material (e.g., marble) has often an important influence on the acquired data since it can cause penetration (Godin et al. 2001; Lichti and Harvey 2002; Guidi et al. 2009b) or bad reflection effects. Moreover, transportability and usability problems arise in certain field campaigns located in remote areas.

For image-based approaches, terrestrial digital cameras must be accurately calibrated, preferably in a controlled lab environment, with a 3D testfield and a bundle adjustment solution with additional parameters to fully compensate for systematic errors (Remondino and Fraser 2006). As no commercial procedure is readily available for automated markerless tie point extraction from terrestrial convergent images, the image orientation phase is still highly interactive, although some recent works seem to be promising in terms of both accuracy and automation (Barazzetti et al. 2009). In case of aerial and satellite imagery, more automation is present in the data processing, although the control points still need to be measured manually. For the surface measurement, manual and semi-automated measurements are still much more reliable in particular for complex architectural scenes or man-made objects. For small free-form objects or ornaments rich of details, dense matching techniques can be instead applied to derive dense 3D point clouds (Remondino et al. 2008).

As far as range-based approaches concerns, the first operations performed on the acquired data are possible errors and outliers removal, noise reduction, and holes filling (Weyrich et al. 2004), followed by the alignment (or registration) of the multiple scans (Salvi et al. 2007). The registration phase is quite straightforward although the identification of homologous points between the overlapping point clouds is still fully interactive unless some targets are placed in the surveyed scene.

Once a point cloud (i.e., unstructured data) is available, a polygonal model (i.e., structured data) needs to be generated to produce the best digital representation of the surveyed object or scene. For architectural scenes and objects, generally described with sparse point clouds, a segmentation and structuring phase is necessary before producing a mesh model. Dense point clouds derived with automated image matching methods or measured with range sensors can be directly converted into meshes, following some possible editing and cleaning. Then some repairing to close holes, fix incorrect faces, or non-manifold parts are often demanding (and time-consuming). Those errors are visually unpleasant, might cause lighting blemishes due to the incorrect normals and the computer model will also be unsuitable for reverse engineering or physical replicas. Moreover, over-sampled areas should be simplified while under-sampled regions should be subdivided. Finally, photo-realism, defined as having no difference between a view rendered from the model and a photograph taken from the same viewpoint, is generally required and achieved with the texture mapping phase, e.g., projecting one or more images (or orthophotos) onto the 3D geometry. Generally, problems might rise from the time-consuming image-to-geometry registration or because of variations in lighting, surface specularity, and camera settings. Often the images are exposed with the illumination at imaging time, but it may need to be replaced by illumination consistent with the rendering point of view and the reflectance properties (bidirectional reflectance distribution function) of the object (Lensch et al. 2003). High dynamic range (HDR) images might also be acquired to recover all scene details (Reinhard et al. 2005) while color discontinuities and aliasing effects must be removed (Debevec et al. 2004; Umeda et al. 2005; Callieri et al. 2008).

The ability to easily interact with a huge 3D model is a continuing and increasing problem, in particular with the new demand of sharing and offering online and real-time visualizations. Indeed, model sizes (both in geometry and texture) are increasing at faster rate than computer hardware and software advances, and this limits the possibilities for interactive and real-time visualization of the 3D results. Due to the generally large amount of data and its complexity, the rendering of large 3D models is done with a multi-resolution approach displaying large textured meshes with different levels of detail and simplification approaches (Luebke et al. 2002; Cignoni et al. 2005; Dietrich et al. 2007).

The integration between the different data sources was essential to overcome some limits of each method and to have as result the complete geometric and appearance information about materials and techniques used to build the heritage.

In Fig. 8, some results of the geometric and appearance modeling are reported with also examples of multi-spectral data. The final photo-realistic 3D model with its multi-spectral layers is now a fundamental basis for conservation and restoration policies to help the local superintendence in preserving the vanishing of the frescoes and correctly restoring the damaged areas.

Open image in new window

When Vesuvius erupted on 24 August A.D. 79, it engulfed the two flourishing Roman towns of Pompeii and Herculaneum, as well as the many wealthy villas in the area. These have been progressively excavated and made accessible to the public since the mid-nineteenth century. The vast expanse of the commercial town of Pompeii contrasts with the smaller but better-preserved remains of the holiday resort of Herculaneum, while the superb wall paintings of the Villa Oplontis at Torre Annunziata give a vivid impression of the opulent lifestyle enjoyed by the wealthier citizens of the Early Roman Empire (http://whc.unesco.org).

The large and complex Roman Forum (approximately 150 ×80 m large with more than 300 scattered archaeological finds on the ground) was digitally reconstructed (Fig. 9a) integrating aerial images (1:3,500 scale, 5 cm GSD), TOF terrestrial laser scanning (1.2 Bil points), close-range images (ca. 5,000), and GPS data for the absolute geo-referencing (Guidi et al. 2009a). The geometric resolution of the 3D data spans from some 20 cm (from the aerial data) down to few millimeters (3D models of relieves derived with terrestrial photogrammetry), with an intermediate LOD given by the TOF range data. Some objects of particular archaeological interest were geometrically modeled in high resolution (Fig. 9b) by means of advanced image matching method (Remondino et al. 2008).

Open image in new window

The entire 3D model of the forum was afterward linked to the existing superintendence archaeological databases (Fig. 9c). The relationship database–3D model was implemented in two ways: (a) from the geometrical 3D data to the archaeological 2D data, for explaining historical and conservation details of a specific artifact in the forum, and (b) from a specific document or philological detail to its corresponding location in the 3D space (Manferdini et al. 2008).