Abstract
Safeguarding and conservation of cultural heritage is an important issue that requires both theoretical knowledge and practical implementation to effectively protect this valuable asset. In recent years, new technologies have enabled the development of advanced monitoring and control systems that can provide more precise and timely information about the condition of heritage buildings. Additionally, these systems can collect and analyze a large amount of data, allowing decision-makers to make informed decisions regarding maintenance and other management processes. With a situational awareness model proposed for heritage building conservation as a starting point, this work outlines how the design and implementation of a cyber-physical system to support conservation processes can be done. The model combines decisional processes that involve both humans and automated systems and can be used as a guideline for the realization of decision support systems for the management of heritage buildings. The paper provides a case study describing the steps for the realization of a cyber-physical system for automated monitoring and control at the Royal Palace of Carditello, a prominent cultural attraction in Italy.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Introduction
Heritage Buildings (HBs) are structures that have been around for a long time and are of cultural or architectural significance. Therefore, safeguarding and conservation of cultural heritage is an important problem whose solution requires both theory (Feilden 2007; Mehr 2019) and good practices (Mekonnen et al. 2022).
The protection of the world’s cultural and natural heritage has received a great impetus since 1972 when the UNESCO Paris Convention established its goals. However, HBs around the world are still confronted with the unending problem of unsatisfactory maintenance management (Akasah et al. 2011; Adegoriola et al. 2021). It has been recently observed that, although the HBs are lucrative assets able to generate millions of jobs and billions of euros in revenue yearly, there is still a need to devote more attention to their preservation (Otero 2021). What appears to be often overlooked is the issue of conservation (Woodward and Heesom 2019), a problem that has many facets.
An important research area related to the issue of conservation is the condition monitoring of HBs by means of sensor networks (Ceriotti et al. 2009; Shen et al. 2022). On the one hand, there is a growing interest in structural health monitoring as a knowledge-based assessment tool to quantify and reduce uncertainties regarding their structural performance (Lorenzoni et al. 2016). Sensors placed at critical points in the structure of HBs enable the detection of anomalies due to various causes such as earthquakes, deterioration of stone or cement, structural damages, pollution, and anthropic activities. On the other hand, preserving works of art usually displayed or maintained in HBs requires appropriate technological solutions (D’Amato et al. 2012; Landi et al. 2022) to support the roles responsible for their state of preservation. This paper aims to contribute to HBs management by proposing a situational awareness (SA) model that integrates human and automatic decision processes. This model is used as a reference for the design of Human-in-the-Loop Cyber-Physical Systems and offers new opportunities to improve monitoring, control, maintenance, and other management processes related to HBs. The case study discussed in the paper describes the cyber-physical system implemented during the realization of the RE-SEMIRTO project at the Royal Palace of Carditello, a major cultural attraction located in Campania, Italy.
The paper is organized as follows. The next section reviews the theoretical background, first discussing the foundations of Intelligent Building (IB) and Heritage Building Maintenance Management (HBMM) and then evaluating the state of the art of edge computing and cyber-physical systems. Section 3 first presents the proposed model of SA that will guide the design and implementation of a management system for HBs; then, a discussion about the goal-directed task analysis (GDTA) used to identify SA requirements and develop solutions to meet user needs follows. Section 4 describes the edge-cloud architecture of the Cyber-Physical System (CPS) for the monitoring and control of HBs. The case study in Sect. 5 presents the system implemented at the Royal Palace of Carditello (San Tammaro, Caserta–Italy), showing the technological solutions used to improve the knowledge base necessary for the conservation of the HBs and the exposed artworks. Section 6 discusses the benefits of our approach to the management of HBs highlighting the contribution concerning the state of the art. The conclusions resume the findings and discuss some limitations that require additional research.
2 Literature review
The literature related to the monitoring of historic buildings is mainly rooted in the research streamlining of IB and HBMM. Many concepts, models, and technologies introduced for IB apply to the maintenance management of HBs buildings (Purwantiasning and Bahri 2017). However, the characteristics of the monumental heritage, and the new technologies, outline a scenario in which additional lines of research and next-generation systems make it possible to improve the efficiency and effectiveness of processes aimed at the monitoring and control of HBs. The theoretical background and the role of edge computing and cyber-physical technologies are the foundation for developing our approach.
2.1 Theoretical background
In the past few decades, the research area of IB has developed from a debate concerning its foundations. There is a multiplicity of definitions of an IB, and each reflects particular aspects that research communities, institutions, or professionals wish to explore (Clements-Croome 2004; Wong et al. 2005). As science and technology advance, breakthroughs aimed at improving IB performance are possible, particularly by leveraging state of art technologies applied to smart building management. Recent works focus on the following aspects: improvement of automation control systems (Yu 2021); artificial intelligence applied to the preventive conservation of HBs (Prieto et al. 2019) and to intelligent decision-making systems in IB (Zhao 2022).
Even if many research results and best practices adopted to manage IB can be applied to different types of buildings, the management of HBs requires special attention due to the long-term goals of heritage conservation and promotion. Indeed, specific practices in heritage conservation and promotion strategies, such as the adaptive reuse of HBs and the conservation of works of art are necessary. In addition, the role of tourism is an important part of HBs management and can influence the decision-making process regarding their conservation and valorization (Biderman and Bosak 1997; Jamieson 2019). Finally, there is a need for a rethinking of consolidated practices and concepts to consider the new scenario of Sustainable Development Goals (Lerario 2022).
The review presented by Adegoriola et al. (2021). highlights the role of HBMM in ensuring sustainable maintenance of HBs aiming at reducing cases of dilapidation, demolition, and loss of social and economic benefits. The authors propose a classification scheme where five broad categories concerning research in HBMM are identified: (1) Decision-making frameworks in HBMM, (2) Integration of digital technologies and HBMM, (3) Condition assessment and maintenance practice, (4) Sustainability and HBMM, and (5) Emerging research trends in HBMM.
In the literature, it is possible to find only a few works that refer to SA theory applied in the IB or HB research. In the paper by Giallonardo et al. (2019), an ontology-based approach to provide runtime models of physical entities in context-aware reactive systems is proposed and validated using a case study of IB for cultural heritage preservation. Context awareness also helps to improve recommendation systems. The work presented in Casillo et al. (2023) describes a general overview of context analysis techniques in recommender systems and discusses some challenging applications aiming at the improvement of the interaction between human beings and artistic-cultural heritage.
An application of GDTA (Endsley and Jones 2004) to obtain the key goals and SA requirements of tasks is discussed in (Irizarry and Gheisari 2014) to solve safety and facility management problems in the construction sector. However, what is less clear in the literature is how SA models and related GDTA provide orientation to developers for the implementation of CPSs to improve the management of HBs. To fill this gap, we propose an SA model that combines decisional processes made by humans and automated systems and show how this model can be used as a guideline for the realization of CPSs to support the management of HBs.
2.2 Technologies for heritage building management
The last decade has seen the rise of smart objects as key components in the implementation of sensor networks. A smart object (SO) is essentially a small micro-electronic device that consists of a communication device, typically a low-power radio, a small microprocessor, and a sensor or actuator (Vasseur and Dunkels 2010). In Kortuem et al. (2010) a SO is defined as an autonomous physical/digital entity augmented with sensing, processing, and network capabilities; furthermore, it carries application logic. A smart object can sense, actuate, log, and interpret what is occurring within itself or in the world in which it is immersed, and can interact with people and other smart objects (Liu and Baiocchi 2016). Figure 1 shows the evolution path from sensors to smart objects.
The presence of a processor as a component of a SO affects the distributed computing model. In the model known as edge computing, the “edge” is defined as any computing and network resources along the path between data sources and cloud data centers (Shi et al. 2016). In this model, data processing occurs as close as possible to where the data are generated, improving response times and saving on bandwidth.
The survey on edge computing for the Internet of Things (Yu et al. 2018) compares the characteristics of IoT, edge computing, and cloud computing proposing a three-layer architecture (IoT devices, intelligent gateway edge computing, and cloud servers) of edge computing-based IoT. Focusing on the transmission, storage, and computation characteristics, the authors highlight how the integration of IoT, and edge computing improves network performance.
An important aspect concerning the acquisition of data of different types and formats in a wireless sensor network (WSN) is data ingestion. Utilizing the proper tools, it is possible to capture, import, clean, filter, manipulate, merge, store, and export data of various formats from many sources having different capabilities of producing data and different transmission rates. A comparative study of data ingestion tools is presented in Sharma et al. (2021).
Research on systems for HBs management takes advantage of the opportunities made available by the advancement of technology. One of the first works that used a WSN to monitor the internal and the external state of an HB is presented in Ceriotti et al. (2009). The authors describe how custom hardware and sensors efficiently capture a high volume of vibration data, using them to detect the potential deformation of a medieval tower. Open-source hardware can also be used for monitoring purposes. The study presented by Mesas-Carrascosa et al. (2016) shows how a network of 15 microclimatic stations was developed using open-source hardware at the Mosque-Cathedral of Córdoba.
Recent research focuses on designing and implementing cyber-physical systems specialized in monitoring and controlling buildings. Essentially, a cyber-physical system (CPS) consists of two parts, a “physical” one that is a selected aspect of the real world and a “cyber” one that is the corresponding virtual representation (Wu et al. 2011). These two parts communicate and influence each other to reach a goal such as the execution of an automated production process or monitoring a building. Typically, communication from the physical system to the virtual system reports real-world status messages (captured by sensors) while reverse communication carries command messages that are executed by actuators. Examples of applications of this technology for building efficiency include Smart Buildings based on IoT and CPS technologies (Bakakeu et al. 2017), and a kind of Cyber-Physical-Social Eco-Society oriented to the current generation of museum visitors, who are influenced by their exposure to modern technology (Nisiotis et al. 2020).
Smart sensor systems have employed classic and modern Machine Learning (ML) algorithms to develop sophisticated models for sensing applications. These models can draw from multiple sensing modalities to gain a comprehensive understanding of the system being monitored. Ha et al. (2020) describes how the sensor technologies are coupled with ML “smart” models and how these systems achieve practical benefits. Afshan and Rout (2021) highlights how ML has the power to handle different challenges associated with IoT data and can be deployed effectively as they require minimal human intervention.
3 The role of situational awareness
Since its introduction, SA has been applied to various application domains. As outlined by Endsley (Endsley 1995; Endsley and Jones 2004), SA is defined as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future”. SA implies that when we need to accomplish a task or reach a goal, we must be aware of our surroundings. The model of SA and dynamic decision-making consists of three distinct levels:
- Level 1:
-
Perception of the elements in the environment.
- Level 2:
-
Comprehension of the current situation.
- Level 3:
-
Projection of future status.
The SA theory provides an effective approach to monitoring, controlling, and HB management. Inspired by the Endsley model (Endsley 2017), the model shown in Fig. 2 is our proposal for SA and the related decision-making processes oriented toward the management of HBs. The model shows how this application domain can take advantage of decisional processes that involve both humans and automated systems.
First, it separates the behavior of automatic systems and humans involved in a decision problem to achieve a goal or perform a job. Although the perception-comprehension-projection scheme is the same in both cases, it is necessary to highlight the peculiarities of the two decision-making processes for each of them. Second, the influence of several stakeholders concerning decisional processes must be taken into account. Third, the role of technology in building automatic systems for monitoring and control purposes must be clearly stated. For what concerns the goals and objectives related to HB management, the main stakeholders are people who have governance roles and people who work in the socio-technical system charged with day-by-day HB management. These people have expectations that must be considered and strongly influence the decisional process oriented to the achievement of organizational purposes. Other stakeholders can influence the decisional processes as well. For example, tourists have various expectations when visiting an HB such as convenient visiting hours, comfortable air conditioning, valorization of the building and the surrounding environment, and easy access to the works of art.
The human decisional process can receive an important contribution from an automatic support system. For what concerns data collection and elaboration, from one side the model shows how heterogeneous data acquired from a sensor network is transformed and organized through a data ingestion phase. The availability of tools to visualize data provides the decision-maker with information on the current situation to make an informed decision downstream of the “human perception, human understanding, human projection” path. On the other side, data output from the data ingestion phase can follow the path “automatic perception, automatic comprehension” feeding a decision support system (DSS) that can trigger a new SA cycle aiming at improving human decision-making. The DSS also receives information stored in the data lake of cloud servers, i.e., building architecture drawings, photos, videos, and other kinds of structured, semi-structured, and unstructured data. In our model, automatic perception has two specific purposes. The first is to execute some transformation and merging operations not strictly necessary for data visualization thus making simpler the data ingestion phase. The second is to organize data in structures that facilitate the subsequent phase of automatic comprehension.
Indeed, there are decisions for HB management that can be taken by an automatic system; typically, they are simple routinary decisions that can be taken autonomously by the system without the intervention of humans. Therefore, the model is enriched by the automatic decision process that follows the path “automatic perception, automatic comprehension, automatic projection”. The choice of smart objects as state-of-the-art technology for the realization of automatic decision-making systems is fundamental to guiding the design of support CPSs which will be discussed in the following section. Smart objects are chosen to implement the necessary sensor network from which environmental variables are collected; they are the fundamental components that allow the feeding of the data ingestion phase. Furthermore, they contribute to the realization of the perception-comprehension-projection modules and influence the quality of the automatic decisional process.
The proposed model suggests that an integrated decision-making system comprises the decision-making processes of human beings together with decision-making processes taken from automatic systems. Table 1 shows the capabilities and strengths of human and automatic systems. Essentially, humans can solve problems and make decisions in complex and changing scenarios where uncertainty makes it difficult to pursue decisions based on incomplete information. On the other hand, there are simpler decisions that can automatically be made based on the analysis of the situation starting from measurements of environmental variables captured by sensors. In these circumstances, the strengths of automatic systems refer to the continuous assessment of the situation and the ability to respond in real-time in an emergency or dangerous situation. It is worthwhile to highlight that there are strengths of automatic decision systems such as “real-time reaction to known environmental stimuli” or “monitoring of variables undetectable by humans” difficult or unreachable by humans. For example, a crack not visible to the human eye in a load-bearing wall of an HB or the physical-chemical characteristics of a painting is perceivable and measurable by specialized sensors and not by humans.
By following the methodology of GDTA (Endsley and Jones 20042004), a hierarchy of goals, decisions, and related information requirements can be developed starting from goals that a human operator must achieve.
Because our study considers the combination of decision-making processes taken by both humans and automated systems, the GDTA hierarchy has been divided into two basic categories concerning the goals of humans and those of automated systems. This approach is driven by the consideration that combining the capabilities and strengths of humans and automated systems recalled in Table 1 can lead to more complete and better-performing DSSs. Furthermore, explicit categorization facilitates the execution of GDTA to identify SA requirements (Nasser-dine et al. 2021). Figure 3 shows the generic hierarchy that is taken as a reference for the development of the case study.
4 The role of cyber-physical systems
The role that CPSs can play in the realization of HB management systems is based on the features that emerging technologies make available to create advanced monitoring, control, information retrieval, and digital services devoted to various kinds of stakeholders. The SOs, their memorization, processing, and interconnection capabilities, coupled with the use of distributed algorithms are the building blocks to build a CPS that supports the SA decision process. They are key components for the implementation of a smart sensor network and the implementation of the automatic perception, comprehension, and projection subsystems described in Fig. 2.
4.1 Smart objects
From the analysis of the literature about SOs emerges that object intelligence comprises several features: self-awareness, interaction with the surrounding environment, data processing, machine learning, and connectivity. These features can be further specified as follows.
-
1.
Self-awareness. The capability of an SO to recognize itself as an individual entity and its current state. It covers the aspects of sensor/actuator identification to establish an access path to the object, status diagnosis, localization, etc.
-
2.
Interaction with the surrounding environment. This functionality is manifested through the relationships that the SO establishes with elements of the surrounding environment using sensors or actuators.
-
3.
Data processing. Processing activities refer to basic processing, i.e., processing the collected primitive data by filtering, algebraic aggregation, conversion, and encryption, and advanced processing by algorithms for statistical analysis, inference, and prediction.
-
4.
Machine Learning. The presence of embedded microprocessors that have memories and sufficient computing capabilities contributes to realize multi-agent applications for ML.
-
5.
Connectivity. The different communication possibilities are manifested through the various available technologies (Wi-Fi, Ble, LoRa, 4 G/5 G). Proper communication management is ensured in a multi-threaded scenario, supported by the GPIO subsystem connected to the microcomputer.
SOs today can be equipped with a dual-core processor that can perform code for:
-
1.
Dialogue with routers via a WiFi connection and with smart IoT in the WSN via LoRa connections.
-
2.
Dialogue via a BLE connection, for example in the case of oscillometric sensors to create a network of NFC microelectromechanical systems (MEMS) or nanoelectromechanical systems (NEMS).
-
3.
Computing for several functionalities such as first processing of data captured from sensors connected directly via some control IO ports (analog, digital, serial, parallel), direct alarm functionality (actuator), and advanced computations executed by ML algorithms.
A final consideration about SOs refers to the fact that they can be both physical and digital objects. This characteristic makes them ideal for implementing cyber-physical systems in which the cyber part constitutes the virtual representation of a physical system that is to be controlled to achieve one or more purposes.
4.2 The edge-cloud approach to the implementation of cyber-physical systems
In the edge-cloud approach, data computing is done closer to the device or equipment that collects it, rather than sending all the raw data to a centralized data center or cloud. This can significantly reduce latency and improve overall performance. However, the integrated edge-cloud approach can coexist with previous architectural solutions, e.g., cloud computing, to cover the different application scenarios potentially realizable. In this section, we explore the edge-cloud approach to the implementation of a cyber-physical system that can be used to sustain the decision processes of Fig. 2.
Edge where? Compute nodes and gateway nodes are more complex devices than IoT sensors because of the large number of functionalities they must provide. Compute and gateway nodes are necessary when the network is primarily made of conventional IOT or Smart IoT nodes with sensing and simple control logic capabilities. This is a typical scenario for smart cities and smart homes where a high number of low-cost nodes suggest the prevalent development of WiFi and BLE telecommunication systems. This scenario is shown in Fig 4 where:
-
a)
The architectural segment “Smart IoT-gateway/router-Cloud” is required when smart IoT components are used as end devices.
-
b)
The architectural segment “IoT-Edge server-Cloud” can be chosen when conventional IoT is used.
When a timely response to changes detected by the sensor is required, IoT smart sensors show limitations due to the absence of an immediate and direct control system. In this case, the SO appears more appropriate and can be used as shown in the architectural segment “Smart object-Edge server-Cloud” of Fig. 4. Examples of applications are:
- Outdoor areas (e.g., climate monitoring, valuable forest areas, marine protected tidal areas, highways, bridges, etc. ).
-Indoor areas where the active or passive anthropogenic risk is considerable (e.g., museums, places of worship, hospitals, convention halls).
Cloud where? Cloud storage is the place where data, information, and knowledge generated by both automated devices and humans are maintained for knowledge management and decision-making purposes. There is no single vision of the cloud in edge-cloud systems as its location may depend on the application scenario. For example, in the case study discussed in Section 5, the edge server should be understood as the bridge that connects the network of SOs and smart IoTs to the cloud with both data ingestion and temporary datastore functions.
The enhancement of the smart function in distributed IoT systems allows the technology associated with the cloud process to be separated into two parts that we could call edge-server and remote cloud. This separation is critical because the edge-server environment allows for completion, with relatively fewer resources than the remote cloud, of:
-
the ML process initiated in the SO nodes
-
the entire ML process in the case of intelligent IoT nodes.
The duality of edge-server and remote-cloud best characterizes the latter in that it leaves the former with the responsibilities of data ingestion and network manager on the edge side of the architecture. The remote cloud, freed from these responsibilities, can exercise the primary role of long-term storage implementing digital services for:
-
data-ingestion for DSS,
-
datalake for other services.
Indeed, given its nature, it can encompass other databases from specialized stations on audio-video, GIS, and other sources.
4.3 Data ingestion
The variety of sensors that can potentially be used in an edge-cloud architecture and their different characteristics require, from a data acquisition perspective, a data ingestion process to acquire data from various sources and load it into a central data repository. This data can then be used for transmission, analysis, ML, and other purposes. In this study, the data ingestion process is critical to arrive at both automated and human-made decisions based on:
-
1.
real-time analysis (done on the edge branch of the architecture) to recognize anomalies and generate notifications, warnings, and alarms;
-
2.
the feeding of a DSS system (done on the cloud side of the architecture) to make available to humans the knowledge they need to make informed decisions.
Automatic data ingestion is developed into two phases. During the first phase, the data detected from various types of sensors connected to a SO are collected and transformed. The SO performs basic functions (capture, import, clean, filter, manipulate, merge, and store data) to increase the quality of collected data. Then, the SO provides functions to export data in various formats for transmission to other SOs or the edge server for further processing. During the second phase, the data ingestion is executed at the edge server or the cloud side of the IoT-Edge-Cloud architecture.
The edge server manages:
-
The data received from an SO in various formats, for example, JSON and CSV, normalizing them.
-
The fusion of data received by several SOs.
-
The generation and visualization of sensor timelines (CFR. Fig. 9).
-
The analysis of incoming anomalies detected and received by the connected SOs, taking care of the delivery of messages sent by SOs.
The cloud manages:
-
The acquisition of big quantities of data either from smart IoT or from edge servers.
-
The feeding of the knowledge base that supports the DSS. For this process, data come in many different shapes and sizes, and their structure depends entirely on the sources from which they were generated and the downstream use cases.
Captured data can be stored in a variety of formats: structured, semi-structured, and unstructured. For example, numeric data for monitoring are presented in the Cloud in Row Key form converted to JSON mode, while diagnostic data are typically unstructured. For what concerns the transmission modality we have two possible choices, streaming and batch.
-
1.
Streaming. An asynchronous message stream containing data from all sensors connected to a SO is triggered periodically (e.g., hourly). As an example of streaming data, we recall those used for telemetry, collected by an SO or IoT-Edge systems, and stored in CSV-type files, JSON, etc. Streaming data can be used to trigger alarms on events, perform timeline studies, or be used as input for specific ML tasks.
-
2.
Batch. Big Data are stored in a set of files that are transferred in bulk and consist of large data sets whose transfer requires high bandwidth. The source files could also be on local servers or other cloud platforms.
Data ingestion initiated by humans complements the automatic data ingestion process. It is necessary because many instruments for observing and measuring phenomena require human intervention to be used. Typically, these instruments produce files of relatively large graphical models, photos, sounds, and movies. The transmission mode of these files to the cloud storage is batch.
5 Case study
The Royal Palace of Carditello, shown in Fig. 5, was designed by the architect Francesco Collecini in 1787 at the behest of Carlo di Borbone and developed under the reign of Ferdinando IV.
After years of neglect and decay, several actions of recovery, protection, preservation, and enhancement have affected this monument. The case study described in this section shows the cyber-physical system implemented during the realization of the RE-SEMIRTO project for the enjoyment, preservation, and enhancement of the HB with the intention of introducing some degree of automatic intelligence to support the management processes.
The first step in designing a CPS implementing the SA model of Fig. 2, is the application of the GDTA to determine the objectives of the operators and the objectives of the automatic systems from which to derive decisions and requirements. For brevity, we report in Fig. 6 only some system goals by dividing them into operator goals (OG) and automated goals (AG) and developing the corresponding sub-goals, decisions, and requirements. For example, goal AG2 (indoor monitoring) is developed into 5 subgoals of which AG2.4 and AG2.5 are about the smooth operation of the processing and communication system. The goal AG2.2 (vibration/oscillation monitoring of artworks) corresponds to decision AD2.2.1 (generate an alarm in the case of sudden movement of the sensor connected to the work of art). Finally, the automatic system SA requirements detail the software functionalities necessary to implement decisions and reach goals.
The second step of the CPS design process is to map the SA model by distributing the automatic intelligence to the nodes in the edge-cloud architecture. To implement the model in Fig. 2, this distribution may depend on the problem to be solved. Among the scenarios we encountered during the development of the case study, two are worth highlighting:
1) Problems that require real-time response where fully automated solutions prevail.
2) Problems that require the skills of humans in which the automated system acts as a support to human decision-makers. In this case, automated behavior and human behavior are combined to improve the decision-making process; the human work is typically long-term analytics done in a much wider time interval than in the previous scenario.
An architectural segment of Fig. 4, describing how the stages of perception-comprehension-projection can be mapped according to the two previous scenarios, is shown in Fig. 7. When real-time problems can be solved with the limited computational resources available in an IoT smart sensor (CFR. Figure 2.c), the entire decision-making process takes place in the IoT smart node. An example is the evaluation of parameters that must remain within predefined thresholds. In this case, the sensor perceives the environmental data and the computational capability of the IoT enables the implementation logic for the understanding phase (e.g. when the data goes beyond the predetermined limits) and projection phase (sending alert messages or triggering feedback actions).
The second scenario is more complex. Examples are the analysis of structures of a historic building or the analysis of a painting. In such cases, the decision-making process combines, as shown in Fig. 2, the following steps: (1) automatic perception, (2) automatic comprehension, (3) human perception, (4) human comprehension, and (5) human projection. For the analysis of paintings described in Sect. 5.2, the automatic process starts sensing environmental variables using various kinds of tools for the capture of photos, videos, and numeric data and continues sending the perceived data to the cloud server. The automatic comprehension is based on data lake technology and follows the server-side ingestion of data from any system at any speed organizing them in a centralized repository of large amounts of structured, semistructured, and unstructured data in native format. Then, software tools and digital services interpret the data (automatic comprehension) and set the scenario for the initiation of the perception-comprehension-projection cycle carried out by the human being. The repository is enriched with data collected from sensors but also with information from other sources that allow the decision-maker to have an adequate knowledge base for understanding the phenomenon he or she is analyzing. The projection phase concerns the identification of a future scenario that may emerge from the data analysis, which, in the case of paintings, could be the need to obtain a restored copy of a painting.
The decision on which nodes to deploy the intelligence (smart IoT, gateway/router, edge server, or cloud server) can be more effective if multi-core processors are used where convenient. For the SOs in our case study, we used dual-core processors of which one dedicated to I/O that can be managed by multi-threaded processes, and another to the execution of software agents running concurrently to implement applicative functions. This approach to the division of computational tasks improves the performances of SOs and makes them critical for the implementation of IoT-edge-cloud architectures.
5.1 Monitoring and control subsystems
Outdoor monitoring is used to assess the effects of air pollution and global climate change (from traffic and urban and industrial settlement) or vibrations/oscillations due to traffic, human activities, or natural phenomena on Cultural Heritage. In Fig. 8 the SO for monitoring is shown. It can be used for outdoor as well as indoor contexts and is equipped with sensors mounted on the board. Moreover, it is connected to many external MEMS-BLE sensors for oscillometric monitoring. This SO consists of 5 sensors and 3 actuators (fan, micro camera, Wi-Fi). Visible is the dual-core processor, 3 families of IoT-class environmental sensors, an IoT sensor control system, the micro camera, and BLE and WiFi antennas for connecting to MEMS and networking. As an example of an actuator, we highlight the fan start/stop mechanism using an NPN-type transistor connected to the GPIO.
The transmission modality managed by the SO is streaming and has the following characteristics:
-
MEMS to SO is on BLE always active transmission channel.
-
SO to edge server is start-stop type driven by ESP32 over WiFi channel.
-
by start-stop type micro camera (triggered periodically or by motion sensors) driven by ESP32 over WiFi channel to the edge server.
The trend of data detected by MEMS sensors is shown in Fig. 9. The dashed frame mark anomalies due to abrupt accelerations of the MEMS sensor in the x, y, and z-axes due to minor displacements of the sensor in three-dimensional space.
Indoor monitoring and control of environmental and thermohygrometric parameters is necessary for the evaluation of the conditions of conservation and enjoyment of works in museums or other contexts of historical and archaeological interest.
5.2 Periodic activities
Periodic activities, carried out by humans with the help of supporting tools, are aimed at diagnosing and assessing the condition of the building and the works of art in it. Photographs, data streaming, and video footage are acquired and transmitted to the cloud repository. The RE-SEMIRTO project includes three kinds of periodic activities:
-
a)
Materials diagnostics for understanding the effects of climatic and atmospheric agents.
-
b)
Analysis of the state of preservation and deterioration of the historic building.
-
c)
Analysis of the state of artworks.
In this way, it is possible to compare the condition of the building and artworks and assess their health or deterioration process with the passage of time.
Figure 10 shows activity (c) for indoor analysis of paintings that involve roles with different professional profiles in a typically multidisciplinary context. The paintings displayed in the “camera oscura” room were analyzed by IR Reflectography, X-Ray Fluorescence, and Reflectance Spectroscopy for the identification and mapping of the different constituting materials (original and restoration pigments) on the pictorial surface, to detect the execution technique and to evaluate persistent degradations or effects due to previous interventions (Spagnuolo et al. 2022).
5.3 Digital services
An important part of the RE-SEMIRTO concerns the creation of Apps aimed at researchers, technicians, guides, and tourists to take advantage of digital services of
-
a)
Dissemination. Implementation of a web-based presentation platform of the results of measurements on cultural heritage, differentiated for the monitoring part, the control part, and the archaeometric and diagnostic surveys. Figure 11 shows an excerpt of detected data available as open data on the platform.
-
b)
Conflict of interest. Definition of Scientific, Technical, and Cultural Pathways of the Real Site of Carditello with the support of appropriate Apps driven by BLE beacons. Figure 12.a) shows the App being used by researchers and technical staff to monitor the values returned by environmental sensors.
-
c)
Participation. Virtual tours to enhance the immersive experience. Virtual tours make visitors aware of ongoing diagnostic investigations and allow visitors to “visit” sites during excavation and artifact analysis/restoration work.
6 Discussion
One of the first papers that applied SA to the construction industry was published by Adamu et al. (2015). The authors present a framework, based on shared SA to enrich co-creation processes and virtual collaboration using cloud-based servers. Their focus is on the design of new constructions rather than safeguarding and conserving HBs, which is the focus of our work. In addition, we use an edge-cloud architecture instead of a traditional cloud architecture allowing us to solve real-time monitoring and control problems. The application of the SA to HBs to evaluate their health through real-time monitoring of key parameters is recent (Xiao 2022). However, this solution does not deepen the decisional process that characterizes SA models as we do in Sect. 3 and does not address the issue of control. Indeed, most of the literature concerning the preservation and conservation of HBs focuses on monitoring several aspects such as:
-
the assessment of their structural integrity using vibration and deformation sensors (Ceriotti et al. 2009; Shen et al. 2022),
-
the interior environments and its artworks using microclimate stations (Mesas-Carrascosa et al. 2016),
This makes sense because monitoring systems are critical to support humans in the decisions they must make to safeguard and conserve cultural heritage and to activate consequential actions.
However, the control component adds new possibilities for managing the HB (Purwantiasning and Bahri 2017).
SA provides a novel way to address the issue of HBs monitoring and control and to frame their decision-making processes in a sound theoretical framework. Our model combines decision-making processes involving humans and automated systems, exploiting their strengths and managing monitoring and control processes. The approach of this study is that of action research (Willis and Edwards 2014), which seeks to solve the current practical problems and simultaneously expand scientific knowledge. Therefore, we chose to identify an SA model that is representative of a family of applications where the asset to be protected is subject to conservation programs. As a result, the application of our model can be replicated in scenarios similar to or even more complex than the case study described in Section 5. For example, the model has been the reference for the design and implementation of the Human-in-the-Loop CPS implemented in the TISMA project (www.tisma.eu) that supports monitoring and control processes of the complex ecosystem of Sant-Agata de- Goti, a historic village in southern Italy (Nota and Petraglia 2023).
From an implementation perspective, the monitoring and control functions are made possible by the implementation of a CPS in which key components are the SOs on which a multiagent system responsible for the various application functions has been implemented. This approach is, in some respects, like that proposed by Fortino et al. (2017) in which agent-based cooperating smart objects are used to develop smart and dynamic IoT systems. For the implementation of automatic decision processes related to HBs management, a CPS has been realized by integrating both the traditional Cloud architecture and the IOT-Edge-Cloud architecture. This solution has several advantages. First, automated decision processes that require real-time or near real-time responses can be implemented by edge devices. Typical functionalities achieved are monitoring and control of the environmental elements with which sensors and actuators interact. Second, the cloud side of the architecture can be used when a greater amount of computational and storage resources is required by applications and there are no urgent real or near real-time constraints. This is the case of DSSs whose knowledge base can be fed either by activities initiated by humans or by automatic mechanisms of data collection. Third, new digital services make it possible to reach dissemination, disclosure, and participation goals.
7 Conclusions
The preservation and conservation of cultural heritage involve the implementation of decision-making processes, taken both by humans and by automated systems. Starting from the proposed SA model for HB conservation, we illustrate how a CPS can be designed and implemented to make automatic decisions and to support different human roles, using the SA model and the GTDA method as design guidelines. The paper highlights the role that SA and CPS can play in the improvement of HB preservation and conservation processes. According to the action research approach, the case has involved collaboration between the researchers from academia and the managers working in the Royal Palace of Carditello so that the implemented CPS and the supported decisional processes have been validated by the managers, especially for the aspects concerning: (a) the benefits of combining the capabilities of humans and the automated components of the CPS that have been summarized in Table 1; (b) the effectiveness of the three-tier edge-cloud architecture to implement the SA model and real-time capabilities for the protection and preservation of historic buildings. Although the case study carried out from the research results presented in the paper met stakeholders’ expectations, there are several limitations that require additional research efforts to address many unresolved aspects. First, a more in-depth study of how risk management (e.g., fire, hydrogeological, pollution, anthropogenic) can be improved through CPS systems is necessary. From a methodological perspective, this requires revisiting the current risk management approaches and the study of a risk ontology in HB management. Another promising research area concerns the development of Cyber-Physical-Social Systems. This is an aspect that is included in our SA model because socio-technical systems, governance, and other stakeholders exercise an important influence on the decisional processes supported by CPSs. However, further study is required to deep how this class of CPSs implements the human-in-the-loop paradigm (Nunes et al. 2015) and how new forms of relationships between humans and ML algorithms, generically called human-in-the-loop ML (Mosqueira-Rey et al. 2022), improve the collaboration between AI models and humans. This is considered increasingly important in the future research due to the knowledge learned by machine learning cannot win human domain knowledge in the near future (Wu et al. 2022). Finally, human-computer interaction is an area of research that is usually considered to improve communication between people and machines so that the performance of related systems and processes can, in turn, be improved. In the scenario of HBs management, the design of human-computer interfaces will draw on studies from other research sectors that are based on context awareness (Bisio et al. 2018), usability (Di Gregorio et al. 2020), and visual pattern design (Fenza et al. 2021).
Data availibility
The dataset used in this paper is publicly available at: https://www.resemirto.it.
References
Adamu ZA, Emmitt S, Soetanto R (2015) Social bim: Co-creation with shared situational awareness. J Inform Technol Construction 20:230–252
Adegoriola MI, Lai JH, Chan EH et al (2021) Heritage building maintenance management (hbmm): a bibliometric-qualitative analysis of literatures. J Build Eng 42:102416
Afshan N, Rout RK (2021) Machine Learning Techniques for IoT Data Analytics. Wiley 3:89–113
Akasah ZA, Abdul RMA, Zuraidi SNF (2011) Maintenance management success factors for heritage building: A framework., WIT Press, pp 653–658. 118
Bakakeu J, Schäfer F, Bauer J, et al (2017) Building Cyber-Physical Systems - A Smart Building Use Case: Foundations, Principles, and Applications, Wiley Telecom, pp 605–639
Biderman AW, Bosak W (1997) Environmental education in protected areas as a contribution to heritage conservation, tourism and sustainable development. In: Springer HBerlin (ed) National Parks and Protected Areas, pp 93–102
Bisio I, Garibotto C, Grattarola A et al (2018) Exploiting context-aware capabilities over the internet of things for industry 4.0 applications. IEEE Netw 32:101–107
Casillo M, Colace F, Conte D et al (2023) Context-aware recommender systems and cultural heritage: a survey. J Ambient Intell Human Comput 14:3109–3127
Ceriotti M, Mottola L, Picco GP, et al (2009) Monitoring heritage buildings with wireless sensor networks: The Torre Aquila deployment. In: International Conference on Information Processing in Sensor Networks, pp 277–288
Clements-Croome DJ (2004) Intelligent Buildings: Design Management and Operation. ICE
D’Amato F, Gamba P, Goldoni E (2012) Monitoring heritage buildings and artworks with wireless sensor networks. In: 2012 IEEE Workshop on Environmental Energy and Structural Monitoring Systems (EESMS), pp 1–6
Di Gregorio M, Nota G, Romano M, et al (2020) Designing usable interfaces for the industry 4.0. In: AVI ’20: Proceedings of the International Conference on Advanced Visual Interfaces, pp 1–9
Endsley MR (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37:32–64
Endsley MR (2017) Situational awareness, publisher, chap Toward a theory of situation awareness in dynamic systems
Endsley MR, Jones DG (2004) Designing for situation awareness–an approach to user-centered design, 2nd edn. CRC Press
Feilden B (2007) Conservation of historic buildings. Routledge
Fenza G, Loia V, Nota G (2021) Patterns for visual management in industry 4.0. Sensors 21(19)
Fortino G, Russo W, Savaglio C et al (2017) Agent-oriented cooperative smart objects: from iot system design to implementation. IEEE Trans Syst Man Cybernet 48:1–18
Giallonardo E, Poggi F, Rossi D, et al (2019) Context-aware reactive systems based on runtime semantic models (s). In: International Conference on Software Engineering and Knowledge Engineering
Ha N, Xu K, Ren G, et al (2020) Machine learning-enabled smart sensor systems. Advanced Intelligent System
Irizarry J, Gheisari M (2014) Situation awareness (SA), a qualitative user-centered information needs assessment approach. Int J Construct Manag 13:35–53
Jamieson W (2019) Overtourism management competencies in asian urban heritage areas. International Journal of Tourism Cities
Kortuem G, Kawsar F, Fitton D et al (2010) Smart objects as building blocks for the internet of things. Internet Comput IEEE 14:44–51
Landi E, Parri L, Moretti R, et al (2022) An iot sensor node for health monitoring of artwork and ancient wooden structures. In: IEEE International Workshop on Metrology for Living Environment (MetroLivEn), pp 110–114
Lerario A (2022) The role of built heritage for sustainable development goals: from statement to action. Heritage 5:2444–2464
Liu X, Baiocchi O (2016) Moving from sensors to smart objects. Revista de Tecnologia da Informação e Comunicação 6:1–5
Lorenzoni F, Casarin F, Caldon M et al (2016) Uncertainty quantification in structural health monitoring: applications on cultural heritage buildings. Mech Syst Signal Process 66:268–281
Mehr SY (2019) Analysis of 19th and 20th century conservation key theories in relation to contemporary adaptive reuse of heritage buildings. Heritage
Mekonnen H, Bires Z, Kassegn B (2022) Practices and challenges of cultural heritage conservation in historical and religious heritage sites: evidence from North Shoa Zone, Amhara Region. Ethiopia, Heritage Science, p 10
Mesas-Carrascosa FJ, Verdú Santano D, Meroño de Larriva JE, et al (2016) Monitoring heritage buildings with open source hardware sensors: A case study of the mosque-cathedral of Córdoba. Sensors 16(10)
Mosqueira-Rey E, Hernández-Pereira E, Alonso-Ríos D et al (2022) Human-in-the-loop machine learning: a state of the art. Artificial Intell Rev 56(4):3005–3054
Nasser-dine A, Moïse A, Lapalme J (2021) Does explicit categorization taxonomy facilitate performing goal-directed task analysis? IEEE Transactions on Human-Machine Systems PP:1–11
Nisiotis L, Alboul L, Beer M (2020) A prototype that fuses virtual reality, robots, and social networks to create a new cyber-physical-social eco-society system for cultural heritage. Sustainability 12:645
Nota G, Petraglia P (2023) The situational awareness approach to the design of cyber-physical systems. submitted to ACM Transaction on Cyber-Physical Systems
Nunes D, Zhang P, Sá Silva J (2015) A survey on human-in-the-loop applications towards an internet of all. IEEE Commun Surveys Tutorials 17:1–1
Otero J (2021) Heritage Conservation Future: Where We Stand, Challenges Ahead, and a Paradigm Shift. Global Challenges 6(1)
Prieto AJ, Ortiz R, Macias J, et al (2019) Artificial intelligence applied to the preventive conservation of heritage buildings, CRC Press, pp 245–249
Purwantiasning A, Bahri S (2017) An Application of Smart Building Concept for Historical Building Using Automatic Control System. Case Study: Fatahillah Museum. International Journal of Built Environment and Scientific Research 1(2):115–122
Sharma G, Tripathi V, Srivastava A (2021) Recent trends in big data ingestion tools: A Study, Springer Singapore, pp 873–881
Shen L, Yang B, Yang Y et al (2022) Real-time monitoring for monolithic movement of a heritage curtilage using wireless sensor networks. Buildings 12:1785
Shi W, Cao J, Zhang Q et al (2016) Edge computing: vision and challenges. IEEE Internet Things J 3(5):637–646
Spagnuolo A, Vetromile C, Masiello A et al (2022) RE-SEMIRTO project: an innovative network of wireless sensors for microclimate monitoring on the Royal Palace of Carditello. J Phys 2204(1):012076
Vasseur JP, Dunkels A (2010) Interconnecting smart objects with IP: The next Internet. Morgan Kaufmann
Willis J, Edwards C (2014) Theoretical foundations for the practice of action research. Models, methods, and examples, Action research, pp 21–43
Wong JK, Li H, Wang SW (2005) Intelligent building research: a review. Automation Construct 14(1):143–159
Woodward A, Heesom D (2019) Implementing HBIM on conservation heritage projects: Lessons from renovation case studies. International Journal of Building Pathology and Adaptation
Wu FJ, Kao YF, Tseng YC (2011) From wireless sensor networks towards cyber physical systems. Pervasive Mobile Comput 7:397–413
Wu X, Xiao L, Sun Y et al (2022) A survey of human-in-the-loop for machine learning. Future Generation Comput Syst 135:364–381
Xiao F (2022) Research on ancient building health monitoring technology based on real-time situation awareness. In: Third International Conference on Computer Science and Communication Technology (ICCSCT 2022), pp 1457–1462
Yu H (2021) Application and design of intelligent building automation control system. J Phys 2074:012077
Yu W, Liang F, He X et al (2018) A survey on the edge computing for the internet of things. IEEE Access 6:6900–6919
Zhao W (2022) Applications of intelligent artificial and communication technologies in intelligent buildings. Highlights Sci Eng Technol 10:156–162
Acknowledgements
This work was carried out as part of the RE-SEMIRTO project funded by the Campania region (Italy). We thank the Carditello Real Site Foundation for their willingness to provide data and images of the real site.
Funding
Open access funding provided by Università degli Studi di Salerno within the CRUI-CARE Agreement.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Nota, G., Petraglia, G. Heritage buildings management: the role of situational awareness and cyber-physical systems. J Ambient Intell Human Comput 15, 2227–2239 (2024). https://doi.org/10.1007/s12652-023-04750-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-023-04750-2