Abstract
The conservation of historic structures requires detailed knowledge of their state of preservation. Documentation of deterioration makes it possible to identify risk factors and interpret weathering mechanisms. It is usually performed using non-destructive methods such as mapping of surface features. The automated mapping of deterioration is a direction not often explored, especially when the investigated architectural surfaces present a multitude of deterioration forms and consist of heterogeneous materials, which significantly complicates the generation of thematic decay maps. This work combines reflectance imaging and supervised segmentation, based on machine learning methods, to automatically segment deterioration patterns on multispectral image composites, using a weathered historic fortification as a case study. Several spectral band combinations and image classification techniques (regression, decision tree, and ensemble learning algorithmic implementations) are evaluated to propose an accurate approach. The automated thematic mapping facilitates the spatial and semantic description of the deterioration patterns. Furthermore, the utilization of low-cost photographic equipment and easily operable digital image processing software adds to the practicality and agility of the presented methodology.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Architectural heritage possesses outstanding value while concomitantly comprises a fundamental manifestation of sociocultural identity. The historic build environment is a vital aspect of a place's culture, history, and landscape, which necessitates measures to ensure its preservation through time. However, environmental pressures and anthropogenic factors cause constant alterations and impose significant risks. Planning appropriate and compatible conservation and restoration interventions to tackle the deterioration of historic structures requires a comprehensive knowledge of the preservation state. Thus, the need for the historic structures' recording emerges, which will provide the detailed information needed to support required preservation interventions.
Recording the preservation state of a historic structure is a crucial prerequisite for pathology diagnosis. Documenting in detail the condition of the structure's elements is the first step towards qualitatively interpreting its condition and identifying mechanisms of deterioration. Therefore defining the data recording techniques that will provide rich and suitable information about the extent and forms of deterioration is essential for condition documentation. To the greatest extent possible, recording should be non-destructive, meaning that it should encompass those nonintrusive inspection and sensing techniques that do not cause further damage to, nor impair the future usefulness of the structure and the historic materials.
Mapping is widely recognized as an effective non-destructive method useful for condition documentation and can be applied to all materials at different scales. It registers information about the surface patterns of historic structures that can be later analyzed through computational systems. Mapping is frequently performed as a manual process in a computer-aided design (CAD) or geographic information system (GIS) environment using as background color photos.
Progress in automated mapping for historic structures has primarily concentrated on identifying and classifying building elements, materials, and additionally deterioration as a binary concept—considering the presence and absence of deterioration solely. The segmentation algorithms that have been considered are mainly based on dimensionality reduction, unsupervised clustering, and deep learning approaches, occasionally considering spectral bands at the infrared range.
1.1 Aims and scope
This work delves into the fields of imaging science and pattern recognition to identify a novel and accurate methodology for classifying different deterioration forms on historic structures. Reflecting on the potential of multispectral imaging and learning-based image analysis for defect detection, the classification of multispectral composites synthesized from reflectance images captured at the visible (RGB), near-ultraviolet (NUV), near-infrared (NIR), and thermal infrared (TIR) spectra, with supervised segmentation methods based on random decision trees, ensemble learning, and regression algorithmic implementations, is thoroughly evaluated.
1.2 Article structure
This article is structured into six sections. Section 2 delivers an overview of the background for the presented work and discusses the related research. Section 3 describes the methodology followed, including the instrumentation, data collection and preparation, algorithmic implementations, and approaches followed to evaluate the segmentation results. Section 4 presents the application and results for the case study of a historic fortification, while Sect. 5 discusses the accuracy and interpretation of the results. The concluding remarks are presented in Sect. 6.
2 Background and related work
Architectural surfaces of historic structures are subjected to continuous alterations due to exposure to environmental conditions, microorganisms, pollution, anthropogenic damages; their susceptibility to decay also depends on (incompatible) conservation interventions of the past and the inherent characteristics of historic materials. Particularly when several different materials are present (such as in masonry structures), the architectural surfaces consist of an intricate mosaic of deterioration forms. Consequently, documentation methods for describing these complicated conditions in a non-destructive way become pertinent and often necessary.
2.1 Mapping the preservation state
Mapping is a valuable non-destructive method as it facilitates the description, registration, and quantification of the, often overlapping, multitude of surface patterns on historic structures. When performed in a digital, computerized manner, it produces spatial information, entities with geometric attributes that can be correlated, compared, used to produce statistical information, and allow for the attribution of semantic data about the characteristics of materials and their decay. Traditionally, mapping is a technique manually performed inside CAD and GIS platforms by describing the shape of surface patterns and organizing them into thematic layers [1,2,3,4,5,6]. An alternative way of mapping deterioration is the visualization of damage levels/indexes, which can be either accomplished directly or indirectly by analyzing the mapped deterioration patterns [7,8,9,10,11,12,13,14,15]. The additional spatial annotation of lithotypes facilitates the association between materials and alteration [1, 11,12,13, 15,16,17,18,19,20,21,22,23]. Mapping supports the interpretation of weathering phenomena when combined with data from non-destructive testing (NDT) [8, 9, 17,18,19,20, 22, 24, 25], laboratory mineralogical, chemical and physical characterization [15, 18, 19, 23, 26], and environmental measurements [8, 12, 18, 19, 25, 27].
2.2 Generating base-maps for deterioration mapping
Mapping is typically a photo-based approach where a color image, an orthorectified image, or an orthoimage-mosaic is used as a base-map for designing the geometrical shape of surface patterns [27]. The metric (accuracy, scale-dependent spatial resolution) and chromatic quality of this background are essential for identifying deterioration [25, 28, 29]. Thus, acquiring suitable images is crucial for successful deterioration mapping. However, not only true color images have been considered as base maps, but also images captured at portions of the electromagnetic spectrum beyond the visible.
2.3 Multispectral imaging and data complementarity
The reciprocity of mapping and infrared reflectance imaging—especially thermography—has often been considered essential for detecting weathering on historic structures [5, 20, 24, 30, 31]. Besides, thermography is being extensively used in built heritage structural diagnostics [32,33,34] and has also been explored to detect different historic materials on building façades [35, 36]. The additional inclusion of NIR reflectance images enhances the identifiability of deterioration, mainly when there is a presence of vegetation and biogenic crusts, which present vastly different near-infrared reflectance characteristics compared with construction materials [37, 38]. However, the decision to include recorded data from multiple bands comes with the realization that suitable sensing techniques have to be selected.
Spectral collection in the infrared is connected with various sensing techniques that depend on the wavelength choice. Detection in the wavelength range between 400 and 1100 nm has been performed with multispectral configurations that involve multiple single-band cameras recording at 4–12 different narrow spectral bands. The resolution of these instruments is usually low, and the collected imagery has to be meticulously checked to correct sensors' errors [39, 40]. The introduction, or rather repurposing, of commercial digital single-lens reflex (DSLR) cameras with charge-coupled device (CCD) and complementary metal–oxide–semiconductor (CMOS)-based detectors, for spectral imaging at the same range, however, provides more affordable and agile solutions that retain the user-friendly features and the interfaces to a wide variety of photographic software and accessories, and have high spatial resolution [41,42,43]. Commercial off-the-shelf (COTS) DSLR camera detectors are generally sensitive in a portion of the NIR range up to 1100 nm, which is cut off by an internal blocking filter. Removing this filter implies that the camera can be used for imaging at a wider than visible range, and external wavelength-specific filters can be utilized. Detection in the long-wavelength infrared (LWIR) range has usually been performed with uncooled microbolometer detectors for building inspections. The spatial resolution of thermography cameras is considerably lower than that of DSLR, and their relative cost is higher. Recently, more affordable thermography camera models have come into the market, including smartphone-adjustable low-resolution instruments. However, these inexpensive cameras provide lower accuracy, which makes them unusable for some applications.
2.4 Digital image processing
The need for more efficient inspection [44] and intelligent identification of conservation needs [45] has led to the adoption of image processing approaches to generate the thematic data needed for deterioration mapping. Digital image processing (DIP) refers to the manipulation of the digital images to extract features and recognize patterns, which, after having acquired the suitable base-maps, can be performed with techniques as simple as thresholding, edge detection, or information reduction to obtain the required results [33, 46,47,48]. However, these approaches still largely depend on the human factor since many parameters have to be tuned differently for each application, and often deterioration patterns have to be identified and extracted one at a time. The current rise of deep learning-based pattern recognition has delivered powerful tools for fully automated detection of deterioration (often through convolutional neural networks), even when a plethora of surface patterns can be observed [49,50,51,52]. Nevertheless, deep-learning implementations require large image datasets to be efficiently trained, which is often impractical for built heritage applications. They may also underperform considering the uniqueness of each heritage asset, many of which present a distinctive mixture of historic materials. Therefore, other more easily executable supervised learning-based approaches are sometimes considered for deterioration detection through classification and regression.
Multiband and multispectral image segmentation for built heritage inspection purposes has been applied via a range of clustering algorithms, some of the most common being maximum-likelihood, minimum-distance, and k-means [36, 37, 39, 48, 53,54,55,56]. However, most of the relevant works aim at segmenting the materials and elements of historic façades, and when deterioration is considered, it is determined as present or not present. To be specific, many works consider the altered and unaltered areas of a historic material as two categories rather than identifying the different deterioration typologies, which is also partly a result of the state of preservation of the heritage assets involved. Alternative multi-sensor approaches, involving terrestrial LiDAR for NIR recording, have been reported to produce high-accuracy thematic mapping results for damaged historic structures [57, 58]. However, they introduce significant instrumentation costs, and require rigorous radiometric calibrations and optimal data gathering conditions.
3 Methods and materials
The rationale behind this work is set on the identified lack of image-based methods for automatic mapping of weathered historic structures. The methods tested aim to tackle the problematics of mapping the preservation state when various surface deterioration forms are present. Instead of following unsupervised segmentation techniques and then interpreting each classified category of weathering-caused alteration, supervised algorithmic approaches are implemented using as input the already identified deterioration categories. Combinations of different spectral band composite images and supervised segmentation algorithms are evaluated to distinguish an optimal solution in terms of accuracy—based on reference data.
Figure 1 depicts the implemented research design in this work. As already highlighted, the quality of available imagery upon which the pattern recognition will be performed is essential for ensuring the accuracy and interpretability of results. Therefore, the workflow starts from acquiring appropriate images and then continues with their radiometric correction. The multispectral composites are digitally synthesized from the band-specific reflectance images and subsequently segmented into deterioration categories following a visual identification of training regions. The results are evaluated with metrics deriving from the field of remote sensing. The output of the deterioration classification can be optionally transferred to an environment appropriate for spatial information management. The principle of using low-cost equipment and software was followed throughout this work as it is an essential factor for the inspection of historic buildings.
3.1 Sensors and data acquisition
The selection of the instruments employed in this work considers the complementarity of data captured at different spectral bands and the flexibility requirements of sensing techniques used for built heritage condition monitoring. Affordable, portable sensors are utilized to obtain the necessary multispectral data that will constitute the background for the deterioration pattern analysis, contributing to a simple to implement methodology. The characteristics of the instrumentation are presented in Table 1. The images are taken with two sensors, an EOS Rebel SL1 (Canon Inc., Tokyo, Japan) digital single-lens reflex camera with an EF-S 18-55 mm f/3.5–5.6 IS II lens, and a FLIR ONE Pro (Teledyne FLIR LLC, Wilsonville, OR, USA) thermographic camera attached to a smartphone. The internal hot mirror filter of the SL1 camera has been removed to allow imaging beyond the visible range. Three low-cost external filters are employed to allow RGB, NUV, and NIR photo shooting. The images are acquired as parallel as possible to the architectural surfaces to avoid occlusions, and with small focal lengths to avoid large distortions that can affect image quality during the resampling phase of distortion correction. Furthermore, the images are acquired under homogeneous illumination conditions and without shadows, improving their radiometric potential and with a steady tripod, thus preventing image blur. Since low-cost sensors are more likely to be affected by noise sources, the camera sensor is checked to estimate the vignetting and background noise levels, and the images are corrected to ensure their quality. The thermographic data are acquired at sets of burst images to increase digitally later their spatial resolution.
3.2 Multispectral data preparation
Pre-processing the imagery data involves the preparation of multispectral image composites for the subsequent segmentation. At first, the radiance images acquired with the SL1 camera are downloaded in the RawDigger (LibRaw LLC, Maryland, USA) software, where the color filter array conversion is reversed to acquire raw radiance images, and RGB images are color balanced. Non-visible spectrum images should also be converted to reflectance images based on pixel values of a reference surface. The uncompressed images are then corrected from distortion [59] in ImageJ2 [60]. Thermal infrared burst mode images acquired with the FLIR ONE Pro camera are used to create high-resolution thermal images [61].
The manual matching of band-specific images is done using the HyperCube software [62] (projective transformation, nearest-neighbor interpolation). Subsequently, the image composites are constructed using different multispectral combinations, as described in Table 2. The images are resampled to match the resolution of all bands, and the sky and ground are trimmed from all multispectral composites to reduce potential misclassifications. The synthesis of the multiband composites also considers the same principle of using low-cost equipment, and thus all composites consist of three bands so that segmentation can be performed in ImageJ2—avoiding the use of commercial specialized spatial analysis software.
3.3 Machine learning-based segmentation of deterioration patterns
The classification of deterioration patterns is performed via a supervised segmentation procedure using the Trainable WeKa Segmentation 3D plugin [63] of ImageJ2. The machine learning-based image segmentation techniques follow decision tree [64], ensemble learning [65], and regression approaches. Specifically, the Random Tree, Random Forest, Fast Random Forest, and LogitBoost classifiers are employed. The supervised approach presupposes the annotation of image regions of interest (ROIs), corresponding to each semantic deterioration category to be segmented, that will train the algorithmic model into providing a semantic classification of the entire image.
The decision tree model is a machine learning algorithm that can be used for both supervised classification and regression problems. A decision tree simply consists of a series of sequential decisions made to reach a specific result of distinct data classes. The classes are mutually exclusive and represented by specific attributes. The learning input, which consists of sets of pixels belonging to known classes, assists the accurate classification of both annotated pixels and not annotated pixels. Each node of the decision tree decides an outcome based on the attribute values and leads either to another node, using an appropriate subtree, or to a leaf, which gives the predicted class of the pixel [66]. The Random Tree classifier is based on a decision tree learning method. Single decision trees are easy to conceptualize but usually suffer from high variance, making them not competitive in terms of accuracy.
A random forest classifier combines ensemble classification machine learning algorithms and decision trees. Each tree classifier is independently generated from the input training data using a random sample like in bagging. When growing a tree, the best possible split is computed for a random subset, instead of always computing the best split for each node. In this way, tree diversity is generated using two ways of randomization. Aggregating predictions make the class prediction of the ensemble. Random forest generally overcomes the accuracy limitations of single decision trees [65, 67].
LogitBoost is a boosting algorithm that performs classification using a regression scheme as the base learner and can handle multi-class problems. It can be seen as a convex optimization; it applies the cost function of logistic regression on a generalized additive model. This classifier determines the appropriate number of iterations by performing efficient internal cross-validation [68].
3.4 Accuracy metrics
The performance of the machine learning classification implementations, and of the different multispectral combinations, are quantitatively evaluated using manually produced degradation maps as the ground truth. Different parameters are used to assess the classification efficiency of the intelligent feature extraction techniques based on accuracy metrics common for thematic mapping. More specifically, the evaluation relies on the precision (fraction of appropriate classification among the classified instances) and F1-score (harmonic mean of precision and sensitivity) calculated for each class (Eqs. 1,2), and on the overall accuracy (Eq. 3)—useful to estimate the overall performance of the classifiers.
where, for each class the TP (true positive), FP (false positive), and FN (false negative) come from the error matrix, a square array of numbers, which express the number of pixels assigned to a particular class in one classification relative to the number of pixels assigned to a particular class in the reference data [69, 70].
3.5 Case study
The historic structure selected as a case study for the application and assessment of the proposed methodology is a fortification in Euboea, Greece (Fig. 2). The Fort of Karababa is an Ottoman fortification constructed in 1684 on the homonymous hill which dominates the Boeotian coast across the city of Chalcis. The construction of the fortress was part of the effort to protect the city of Chalcis from impending Venetian attacks. The architectural style of the fort is more European than Turkish. It is oblong in plan, with a rampart on the north side, three bastions, and a large tower. Several parts of the fortification walls have ancient spolia built-in, while the south part is preserved in poor condition. The weathered masonry surfaces selected for evaluating the methodology are presented in Fig. 3. They are on the north side, and for abbreviation purposes, they have been named A, B, and C, starting with the westernmost (on the north bastion).
4 Results
Following the described methodologies, after the composition of multispectral images was completed (Fig. 4), 60 classifications were performed. The generation of reference maps considered the Illustrated Glossary on Stone Deterioration Patterns [71] as a guide during visual inspection. The observed categories of deterioration were vegetation, moss, black crusts, lichens, missing material (including loss of components, large cracks, and windows), and dampness. These constituted all the categories of surface pathology that altered the surface reflectance characteristics of the masonry façades. Patterns that caused slight geometrical surface alterations, such as minor cracks, superficial cracking due to biogenic deterioration, disintegration, or other shape features induced by material loss, insignificant concerning the considered scale and the reflectance contract comparing with healthy historic materials could not be considered. The thematic comparisons were performed using the full reference maps and not sampled patch areas. Overall accuracy statistics calculated from the confusion matrixes are presented in Table 3. The precision and F1-score results are presented in detail in “Appendix A”.
The deterioration maps produced for all the studied architectural surfaces were of generally high thematic accuracy, especially for classifications performed with ensemble learning-based algorithmic implementations. Furthermore, the inclusion of different spectral bands improved the classification potential, subject to the categories of deterioration present.
5 Discussion
The inclusion of the NIR spectral band fairly improved the classification results for all deterioration forms. The segmentation of a NIR-R-G multispectral image and the Fast Random Forest classifier proved to be the most consistent solution overall (79 ≤ overall accuracy%). Figure 5 presents a comparison between the reference maps and the NIR-R-G composites segmented with the Fast Random Forest. Using NUV reflectance data generally did not provide any improvement to the quality of the classifications. Including the TIR band also did not improve the deterioration patterns' classification. Furthermore, the fusion of visible with thermal data significantly decreased the accuracy of detecting deterioration when dampness was present, which contradicts that thermal images are helpful in detecting moisture on historic masonry, as evident by Fig. 6.
According to the overall accuracy results, the Fast Random Forest classifier was the most accurate learning-based method for deterioration classification for all multispectral images, not including the TIR band (70% < overall accuracy < 87%). Implementing the random tree classifier resulted in more inconsistent and less accurate classifications (60% < overall accuracy < 77%). LogitBoost outperformed the Random Tree classifier.
According to the precision and F1-score values, moss and lichens were the most misclassified surface patterns, even though both random forest approaches improved their classification. The results prove that the distinction among non-deteriorated material, dampness, black crusts/discoloration, and plants is much more easily detectable (and therefore classifiable) than biogenic colonization of any form. Therefore, surface alterations of the historic materials—which alter the reflectance characteristics—can be more accurately mapped using multispectral images in comparison with the deterioration forms that completely cover them as an additional layer.
6 Conclusions
In this work, a novel methodology for the automatic classification of damage on built cultural heritage was proposed that uses low-cost photographic equipment for multispectral data acquisition and supervised machine learning-based image segmentation to map deterioration patterns. It was confirmed that including near-infrared reflectance intensities in the employed methods improved the classification of alterations on the historic masonry façades.
The segmentation of multispectral composites (synthesized with visible and near-infrared reflectance images), with classifiers combining random trees and ensemble learning, performed particularly well even were a high number of surface patterns was present. However, the coexistence of different overlapping categories of biogenic colonization complicated the mapping procedure significantly. It should be highlighted that the accuracy evaluation considered some level of bias since the manually produced reference thematic maps cannot consider the overlapping surface patterns.
The proposed methodology has the limitation that it can map only the pathologies that have been previously recognized through visual inspection (or analytical techniques) because regions of interest have to be annotated to train the intelligent algorithms. However, a crucial advantage is that it produces easily interpretable mapping results, in contradiction to unsupervised methods where each mapped pattern class has to be a posteriori assigned to a deterioration category. Furthermore, there is a clear advantage over deep learning-based methods, that require large image datasets, for rapid monitoring purposes of monumental heritage structures. A direct outlook of the proposed framework is the combination with 3D recording technologies to enhance the capability of detecting and mapping the geometric altering (material loss) of historic monuments.
Availability of data and material
Restrictions apply to the availability of the images used for this project. Images were acquired with the permission of the Ephorate of Antiquities of Euboea and are available from the author only if proper authorization can be obtained from the Ephorate of Antiquities of Euboea.
References
Fitzner B, Heinrichs K (2001) Damage diagnosis on stone monuments–weathering forms, damage categories and damage indices. Acta Univ Carol Geol 45(1):12–13
Inkpen R, Duane B, Burdett J, Yates T (2008) Assessing stone degradation using an integrated database and geographical information system (GIS). Environ Geol 56:789–801. https://doi.org/10.1007/s00254-008-1309-x
Brunetaud X, Luca LD, Janvier-Badosa S, Beck K, Al-Mukhtar M (2012) Application of digital techniques in monument preservation. Eur J Environ Civ Eng 16(5):543–556. https://doi.org/10.1080/19648189.2012.676365
Janvier-Badosa S, Beck K, Brunetaud X, Al-Mukhtar M (2013) Historical study of Chambord Castle: basis for establishing the monument health record. Int J Archit Herit 7(3):247–260. https://doi.org/10.1080/15583058.2011.634959
Mileto C, Vegas F, Lerma JL (2015) Multidisciplinary studies, crossreading and transversal use of thermography: the Castle of Monzón (Huesca) as a case study. In: Rodríguez-Navarro P (ed) Defensive architecture of the mediterranean. Editorial Universitat Politècnica de València, Valencia, pp 405–412
Janvier-Badosa S, Brunetaud X, Beck K, Al-Mukhtar M (2016) Kinetics of stone degradation of the castle of Chambord in France. Int J Archit Herit 10(1):96–105. https://doi.org/10.1080/15583058.2014.971194
Fitzner B, Heinrichs K, Bouchardiere DL (2003) Weathering damage on Pharaonic sandstone monuments in Luxor-Egypt. Build Environ 38(9–10):1089–1103. https://doi.org/10.1016/S0360-1323(03)00086-6
Heinrichs K (2008) Diagnosis of weathering damage on rock-cut monuments in Petra. Jordan Environ Geol 56(3–4):643–675. https://doi.org/10.1007/s00254-008-1358-1
Heinrichs K, Fitzner B (2011) Assessment of weathering damage on the Petroglyphs of Cheonjeon-ri, Ulsan, Republic of Korea. Environ Earth Sci 63(7–8):1741–1761. https://doi.org/10.1007/s12665-010-0828-4
Delgado Rodrigues J (2015) Defining, mapping and assessing deterioration patterns in stone conservation projects. J Cult Herit 16:267–275. https://doi.org/10.1016/j.culher.2014.06.007
Franković M, Novaković N, Matović V (2015) Damage quantification of built stone on Dark Gate (Belgrade, Serbia): sample of damage index application for decay rate evaluation. Environ Earth Sci 73(10):6181–6193. https://doi.org/10.1007/s12665-014-3843-z
Gizzi FT, Sileo M, Biscione M, Danese M, Alvarez de Buergo M (2016) The conservation state of the Sassi of Matera site (Southern Italy) and its correlation with the environmental conditions analysed through spatial analysis techniques. J Cult Herit 17:61–74. https://doi.org/10.1016/j.culher.2015.05.002
Germinario C, Gorrasi M, Izzo F, Langella A, Limongiello M, Mercurio M, Musmeci D, Santoriello A, Grifa C (2020) Damage diagnosis of Ponte Rotto, a Roman bridge along the ancient Appia. Int J Conserv Sci 11(1):277–290
Randazzo L, Collina M, Ricca M, Barbieri L, Bruno F, Arcudi A, La Russa MF (2020) Damage indices and photogrammetry for decay assessment of stone-built cultural heritage: the case study of the San Domenico church main entrance Portal (South Calabria, Italy). Sustainability 12(12):5198. https://doi.org/10.3390/su12125198
Puy-Alquiza MJ, Ordaz Zubia VY, Aviles RM, MaDC S-H (2021) Damage detection historical building using mapping method in music school of the University of Guanajuato. Mexico Mech Adv Mater Struct 28(10):1049–1060. https://doi.org/10.1080/15376494.2019.1629049
McCabe S, Smith BJ, Warke PA (2007) An holistic approach to the assessment of stone decay: Bonamargy Friary, Northern Ireland. Geol Soc Spec Publ 271:77–86. https://doi.org/10.1144/GSL.SP.2007.271.01.09
Delegou ET, Tsilimantou E, Oikonomopoulou E, Sayas J, Ioannidis C, Moropoulou A (2013) Mapping of building materials and conservation interventions using GIS: the case of Sarantapicho Acropolis and Erimokastro Acropolis in Rhodes. Int J Herit Digit Era 2(4):631–653. https://doi.org/10.1260/2047-4970.2.4.631
Adamopoulos E, Tsilimantou E, Keramidas V, Apostolopoulou M, Karoglou M, Tapinaki S, Ioannidis C, Georgopoulos A, Moropoulou A (2017) Multi-sensor documentation of metric and qualitative information of historic stone structures. ISPRS Ann Photogramm Remote Sens Spatial Inf Sci. https://doi.org/10.5194/isprs-annals-IV-2-W2-1-2017
Martínez-Martínez J, Benavente D, Jiménez Gutiérrez S, García-del-Cura MA, Ordóñez S (2017) Stone weathering under Mediterranean semiarid climate in the fortress of Nueva Tabarca island (Spain). Build Environ 121:262–276. https://doi.org/10.1016/j.buildenv.2017.05.034
İnce İ, Bozdağ A, Tosunlar MB, Hatır ME, Korkanç M (2018) Determination of deterioration of the main facade of the Ferit Paşa Cistern by non-destructive techniques (Konya, Turkey). Environ Earth Sci 77(11):420. https://doi.org/10.1007/s12665-018-7595-z
Gulotta D, Toniolo L (2019) Conservation of the built heritage: pilot site approach to design a sustainable process. Herit 2(1):797–812. https://doi.org/10.3390/heritage2010052
Hatır ME, Korkanç M, Başar ME (2019) Evaluating the deterioration effects of building stones using NDT: the Küçükköy Church, Cappadocia Region, central Turkey. Bull Eng Geol Environ 78(5):3465–3478. https://doi.org/10.1007/s10064-018-1339-x
Iandelli N, Coli M, Donigaglia T, Ciuffreda AL (2021) An unconventional field mapping application: a complete opensource workflow solution applied to lithological mapping of the coatings of cultural heritage. IJGI 10(6):357. https://doi.org/10.3390/ijgi10060357
Kilic G (2015) Using advanced NDT for historic buildings: Towards an integrated multidisciplinary health assessment strategy. J Cult Herit 16(4):526–535. https://doi.org/10.1016/j.culher.2014.09.010
Silveira da Costa V, Montagna da Silveira A, da Silva TA (2021) Evaluation of degradation state of historic building facades through qualitative and quantitative indicators: case study in Pelotas Brazil. Int J Archit Herit. https://doi.org/10.1080/15583058.2021.1901161
Lezzerini M, Antonelli F, Columbu S, Gadducci R, Marradi A, Miriello D, Parodi L, Secchiari L, Lazzeri A (2016) Cultural heritage documentation and conservation: three-dimensional (3D) laser scanning and geographical information system (GIS) techniques for thematic mapping of facade stonework of St. Nicholas Church (Pisa, Italy). Int J Archit Herit 10(1):9–19. https://doi.org/10.1080/15583058.2014.924605
Gutiérrez-Carrillo ML, Bestué Cardiel I, Molero Melgarejo E, Marcos Cobaleda M (2020) Pathologic and risk analysis of the Lojuela Castle (Granada-Spain): methodology and preventive conservation for medieval earthen fortifications. Appl Sci 10(18):6491. https://doi.org/10.3390/app10186491
Thornbush MJ, Viles HA (2007) Photo-based decay mapping of replaced stone blocks on the boundary wall of Worcester college. Oxford Geol Soc Spec Publ 271:69–75. https://doi.org/10.1144/GSL.SP.2007.271.01.08
Russo M, Carnevali L, Russo V, Savastano D, Taddia Y (2019) Modeling and deterioration mapping of façades in historical urban context by close-range ultra-lightweight UAVs photogrammetry. Int J Archit Herit 13(4):549–568. https://doi.org/10.1080/15583058.2018.1440030
Jo YH, Lee CH (2014) Quantitative modeling and mapping of blistering zone of the Magoksa Temple stone pagoda (13th century, Republic of Korea) by graduated heating thermography. Infrared Phys Technol 65:43–50. https://doi.org/10.1016/j.infrared.2014.02.011
Napolitano R, Hess M, Glisic B (2020) Quantifying the differences in documentation and modeling levels for building pathology and diagnostics. Arch Computat Methods Eng 27(4):1135–1152. https://doi.org/10.1007/s11831-019-09350-y
Maierhofer C, Röllig M, Krankenhagen R (2010) Integration of active thermography into the assessment of cultural heritage buildings. J Mod Opt 57(18):1790–1802. https://doi.org/10.1080/09500341003703497
Moropoulou A, Labropoulos KC, Delegou ET, Karoglou M, Bakolas A (2013) Non-destructive techniques as a tool for the protection of built cultural heritage. Constr Build Mater 48:1222–1239. https://doi.org/10.1016/j.conbuildmat.2013.03.044
Brooke C (2018) Thermal Imaging for the archaeological investigation of historic buildings. Remote Sens 10(9):1401. https://doi.org/10.3390/rs10091401
Lerma JL (2001) Multiband versus multispectral supervised classification of architectural images. Photogramm Rec 17(97):89–101. https://doi.org/10.1111/0031-868X.00169
Lerma JL (2005) Automatic plotting of architectural facades with multispectral images. J Surv Eng 131(3):73–77. https://doi.org/10.1061/(ASCE)0733-9453(2005)131:3(73)
Lerma JL, Cabrelles M, Akasheh TS, Haddad NA (2012) Documentation of weathered architectural heritage with visible, near infrared, thermal and laser scanning data. Int J Herit Digit Era 1(2):251–275. https://doi.org/10.1260/2047-4970.1.2.251
Armesto-González J, Riveiro-Rodríguez B, González-Aguilera D, Rivas-Brea MT (2010) Terrestrial laser scanning intensity data applied to damage detection for historical buildings. J Archaeol Sci 37(12):3037–3047. https://doi.org/10.1016/j.jas.2010.06.031
Del Pozo S, Herrero-Pascual J, Felipe-García B, Hernández-López D, Rodríguez-Gonzálvez P, González-Aguilera D (2016) Multispectral radiometric analysis of façades to detect pathologies from active and passive remote sensing. Remote Sens 8(1):80. https://doi.org/10.3390/rs8010080
Nocerino E, Dubbini M, Menna F, Remondino F, Gattelli M, Covi D (2017) Geometric calibration and radiometric correction of the MAIA multispectral camera. Int Arch Photogramm Remote Sens Spat Inf Sci. https://doi.org/10.5194/isprs-archives-XLII-3-W3-149-2017
Verhoeven G (2008) Imaging the invisible using modified digital still cameras for straightforward and low-cost archaeological near-infrared photography. J Archaeol Sci 35(12):3087–3100. https://doi.org/10.1016/j.jas.2008.06.012
Falco CM (2009) Invited Article: High resolution digital camera for infrared reflectography. Rev Sci Instrum 80(7):071301. https://doi.org/10.1063/1.3174431
Webb EK, Robson S, MacDonald L, Garside D, Evans R (2018) Spectral and 3D cultural heritage documentation using a modified camera. Int Arch Photogramm Remote Sens Spatial Inf Sci. https://doi.org/10.5194/isprs-archives-XLII-2-1183-2018
Dias IS, Flores-Colen I, Silva A (2021) Critical analysis about emerging technologies for building’s Façade inspection. Buildings 11(2):53. https://doi.org/10.3390/buildings11020053
Marzouk M, Zaher M (2020) Artificial intelligence exploitation in facility management using deep learning. Constr Innov 20(4):609–624. https://doi.org/10.1108/CI-12-2019-0138
Cossu R, Chiappini L (2004) A color image segmentation method as used in the study of ancient monument decay. J Cult Herit 5(4):385–391. https://doi.org/10.1016/j.culher.2004.03.005
Vázquez MA, Galán E, Guerrero MA, Ortiz P (2011) Digital image processing of weathered stone caused by efflorescences: a tool for mapping and evaluation of stone decay. Constr Build Mater 25(4):1603–1611. https://doi.org/10.1016/j.conbuildmat.2010.10.003
Adamopoulos E, Rinaudo F (2021) Documenting the state of preservation of historical stone sculptures in three dimensions with digital tools. In: Del Bimbo A, Cucchiara R, Sclaroff S, Farinella GM, Mei T, Bertini M, Escalante HJ, Vezzani R (eds) Pattern recognition. ICPR international workshops and challenges. Springer International Publishing, Cham, pp 666–673
Hatir ME, Barstuğan M, İnce İ (2020) Deep learning-based weathering type recognition in historical stone monuments. J Cult Herit 45:193–203. https://doi.org/10.1016/j.culher.2020.04.008
Mansuri LE, Patel DA (2021) Artificial intelligence-based automatic visual inspection system for built heritage. SASBE. https://doi.org/10.1108/SASBE-09-2020-0139
Hatır ME, İnce İ, Korkanç M (2021) Intelligent detection of deterioration in cultural stone heritage. J Build Eng 44:102690. https://doi.org/10.1016/j.jobe.2021.102690
Pathak R, Saini A, Wadhwa A, Sharma H, Sangwan D (2021) An object detection approach for detecting damages in heritage sites using 3-D point clouds and 2-D visual data. J Cult Herit 48:74–82. https://doi.org/10.1016/j.culher.2021.01.002
Meroño JE, Perea AJ, Aguilera MJ, Laguna AM (2015) Recognition of materials and damage on historical buildings using digital image classification. S Afr J Sci 111(1/2):1–9. https://doi.org/10.17159/sajs.2015/20140001
Sánchez-Aparicio LJ, Del Pozo S, Ramos LF, Arce A, Fernandes FM (2018) Heritage site preservation with combined radiometric and geometric analysis of TLS data. Autom Constr 85:24–39. https://doi.org/10.1016/j.autcon.2017.09.023
Grilli E, Remondino F (2019) Classification of 3D digital heritage. Remote Sens 11(7):847. https://doi.org/10.3390/rs11070847
Adamopoulos E, Rinaudo F (2021) Combining multiband imaging, photogrammetric techniques, and FOSS GIS for affordable degradation mapping of stone monuments. Buildings 11(7):304. https://doi.org/10.3390/buildings11070304
Sánchez-Aparicio LJ, Del Pozo S, Rodriguez-Gonzalvez P, Herrero-Pascual J, Muñoz-Nieto A, Gonzalez-Aguilera D (2016) Practical use of multispectral techniques for the detection of pathologies in constructions. In: Riveiro B, Solla M (eds) Non-destructive techniques for the evaluation of structures and infrastructure, 1st edn. CRC Press, London, pp 253–271
Pozo-Antonio JS, Puente I, Pereira MFC, Rocha CSA (2019) Quantification and mapping of deterioration patterns on granite surfaces by means of mobile LiDAR data. Measurement 140:227–236. https://doi.org/10.1016/j.measurement.2019.03.066
Kaynig V, Fischer B, Müller E, Buhmann JM (2010) Fully automatic stitching and distortion correction of transmission electron microscope images. J Struct Biol 171(2):163–173. https://doi.org/10.1016/j.jsb.2010.04.012
Rueden CT, Schindelin J, Hiner MC, DeZonia BE, Walter AE, Arena ET, Eliceiri KW (2017) Image J2: ImageJ for the next generation of scientific image data. BMC Bioinform 18(1):529. https://doi.org/10.1186/s12859-017-1934-z
Shepard D (1968) A two-dimensional interpolation function for irregularly-spaced data. Proceedings of the 1968 23rd ACM national conference. ACM Press, London, pp 517–524. https://doi.org/10.1145/800186.810616
HyperCube: Spectral Imagery Analysis Software (2021). https://www.erdc.usace.army.mil/Media/Fact-Sheets/Fact-Sheet-Article-View/Article/610433/hypercube. Accessed 21 June 2021
Arganda-Carreras I, Kaynig V, Rueden C, Eliceiri KW, Schindelin J, Cardona A, Sebastian Seung H (2017) Trainable Weka Segmentation: a machine learning tool for microscopy pixel classification. Bioinformatics 33(15):2424–2426. https://doi.org/10.1093/bioinformatics/btx180
Quinlan JR (1993) C4.5: programs for machine learning, 1st edn. Morgan Kaufmann, San Mateo
Breiman L (2001) Random forests. Mach Learn 45:5–23
Quinlan JR (1996) Learning decision tree classifiers. ACM Comput Surv 28(1):71–72
Zhao Y, Zhang Y (2008) Comparison of decision tree methods for finding active objects. Adv Space Res 41(12):1955–1959. https://doi.org/10.1016/j.asr.2007.07.020
Friedman J, Hastie T, Tibshirani R (2000) Additive Logistic Regression: a Statistical View of Boosting. Ann Stat 28(2):337–374
Congalton RG, Green K (2019) Assessing the accuracy of remotely sensed data: principles and practices, 3rd edn. Taylor & Francis Group, Boca Raton
Das R (2020) Content-based image classification: efficient machine learning using robust feature extraction techniques, 1st edn. Chapman and Hall/CRC, Boca Raton
ICOMOS-ISCS (2008) Illustrated glossary on stone deterioration patterns, 1st edn. Ateliers 30 Impression, Champigny-sur-Marne
Acknowledgements
The author acknowledges the Hellenic Ministry of Culture and Sports/Archaeological Resources Fund. All copyrights to the depicted monuments belong to the Hellenic Ministry of Culture and Sports (law 3028/2002). The fort of Karababa in Chalcis falls within the jurisdiction of the Ephorate of Antiquities of Euboea. The author extends his gratitude to the Ephorate of Antiquities of Euboea for granting permission to capture, reproduce and disseminate for research purposes images of archaeological content regarding the depicted monument.
Funding
Open access funding provided by Università degli Studi di Torino within the CRUI-CARE Agreement. This study was funded by the European Union’s Framework Program for Research and Innovation Horizon 2020 (H2020) under the Marie Skłodowska-Curie Actions (MSCA) COFUND scheme (grant number 754511), and by the foundation Compagnia di San Paolo.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares no conflict of interest. The funders had no role in the study's design, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Adamopoulos, E. Learning-based classification of multispectral images for deterioration mapping of historic structures. J Build Rehabil 6, 41 (2021). https://doi.org/10.1007/s41024-021-00136-z
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s41024-021-00136-z