Abstract
The use of deep generative models (DGMs) such as variational autoencoders, autoregressive models, flow-based models, energy-based models, generative adversarial networks, and diffusion models has been advantageous in various disciplines due to their high data generative skills. Using DGMs has become one of the most trending research topics in Artificial Intelligence in recent years. On the other hand, the research and development endeavors in the civil structural health monitoring (SHM) area have also been very progressive owing to the increasing use of Machine Learning techniques. As such, some of the DGMs have also been used in the civil SHM field lately. This short review communication paper aims to assist researchers in the civil SHM field in understanding the fundamentals of DGMs and, consequently, to help initiate their use for current and possible future engineering applications. On this basis, this study briefly introduces the concept and mechanism of different DGMs in a comparative fashion. While preparing this short review communication, it was observed that some DGMs had not been utilized or exploited fully in the SHM area. Accordingly, some representative studies presented in the civil SHM field that use DGMs are briefly overviewed. The study also presents a short comparative discussion on DGMs, their link to the SHM, and research directions.
Avoid common mistakes on your manuscript.
1 Introduction
Structural health monitoring (SHM) plays a crucial role in ensuring the safety, reliability, and longevity of civil engineering structures by identifying issues at an early stage, optimizing maintenance activities, and enabling informed decision-making (Malekzadeh et al., 2015). SHM typically consists of sensing and instrumentation, data collection, preprocessing, analysis, and evaluation phases, followed by decision-making. It includes various data-driven techniques (Catbas et al., 2013, 2022).
Using SHM systems, e.g., data acquisition components, accelerometers, strain gauges, and other sensors, on every civil structure is not economical. It is widely known that data collection procedures from civil structures can be difficult and expensive, restricting the information flow obtained from structures. Due to the challenges in data collection, the data scarcity phenomenon is a crucial issue in SHM. On the other hand, information loss during the monitoring process is often caused by sensor- or transmission-based errors, making data scarcity more critical. The fact that SHM is composed of data-driven applications increases the importance of the data scarcity issue even further (Luleci & Catbas, 2022).
During the last few decades, the research and development in the civil SHM field have been very progressive due to the increasing use of machine learning (ML) and different deep learning (DL) models to address the challenging problems faced in the field, such as data scarcity (Avci et al., 2021; Luleci et al., 2022). Such models have also been used in interdisciplinary research problems (e.g., seismic damage assessment, building inventory assessment, or post-earthquake recovery models) (Soleimani-Babakamali & Zaker Esteghamati, 2022; Soleimani‐Babakamali et al., 2022; Xu et al., 2021a; Luleci & Catbas, 2023).
ML-based techniques have been a research trend for the last few decades in many SHM applications. Deep generative models, in short DGMs, are generative models with many hidden neural networks that have been highly favored in recent years across various disciplines. They are a powerful way of learning hidden data representations in data distributions and generating new data instances with variations by leveraging the flexibility of deep neural networks.
DGMs can be particularly useful in addressing the data scarcity issue in SHM. For instance, when a non-to-limited amount of labeled or high-quality data is available, DGMs can help overcome this difficulty. They could provide distinct ways to generate data for different goals. Those ways, in general, are data generation (generation only—e.g., for general data needs), lost data reconstruction (e.g., for lost or missing data points), data augmentation (e.g., for improving the class imbalance in classification problems), data domain translation (e.g., for no access to data pairs for classification problems or undamaged-to-damaged domain translation), data denoising and repairing (e.g., for noisy and bad quality data), anomaly and novelty detection (e.g., for data consists of anomalies and novelties), others (e.g., for damage identification, annotation reduction via transfer learning), and various approaches that have yet to be explored (Luleci et al., 2022). These approaches can enhance the performance, robustness, and generalization capabilities of data-driven tools used in SHM applications where data availability is non-to-limited.
2 Deep generative models
Rather than creating a decision boundary in the data distribution for classification purposes, which is the discriminative approach, the generative approach aims to learn how the data distributions are shaped. The deep neural networks are used in DGMs to parametrize the generative models, increasing the model's learning capacity. When the DGMs are trained successfully, they can generate new data points similar to the data points from the unknown distribution.
DGMs generally consist of six members (Bond-Taylor et al., 2022; Ruthotto & Haber, 2021; Tomczak, 2022): autoregressive models (AMs), variational autoencoders (VAEs), flow-based models (FBMs), energy-based models (EBMs), generative adversarial networks (GANs), and lastly diffusion models (DMs). The general concepts of the DGMs are briefly explained in the subsequent paragraphs without getting into mathematics. As a side note, it was observed that while some DGMs have been explored, some others have not been studied in the SHM area during the preparation of this short review communication as of December 2022. It should also be noted that only the representative studies presented in the civil SHM field that use DGMs are briefly overviewed.
Figure 1 illustrates the summary of the mechanisms of deep generative models (Chahal et al., 2020; Weng, 2021). In the figure, x and x’ are, respectively, original and synthetic data; z is the latent variable; y and \(\widehat{y}\) are desired and resulted data instances; C(y,\(\widehat{y}\)) compares the desired and resulted instances and gives a score; f is the invertible transformation function; qφ(z|x) and pφ(x|z) are the probabilistic encoder and decoder; D(x) and G(x) are the discriminator and generator. While likelihood-based models such as VAEs, AMs, FBMs, EBMs, and DMs can be trained stably, training implicit models like GANs can be unstable. In VAE, only the lower bound is provided, and the likelihood function cannot be precisely computed, which is also true for EBMs requiring calculating the partition function. AMs suffer from the sampling process, which makes the inference extremely slow due to the autoregressive manner of generating new data points; however, they are one of the most efficient likelihood models in terms of their structure. EBMs and DMs require to run Monte Carlo for inference, slowing down the generation. Nevertheless, DMs are currently state-of-the-art DGM, demonstrating better generative performance than even GANs (Dhariwal & Nichol, 2021).
2.1 Variational Autoencoders (VAEs)
VAEs were first introduced by Kingma and Welling (2013), which are probabilistic generative models that combine the concepts of autoencoders and variational inference. Autoencoders are neural network architectures that learn to encode and decode data, compressing it into a lower-dimensional latent space. Variational inference is a statistical technique used to approximate complex probability distributions. VAEs introduced a new approach to unsupervised learning by leveraging the power of neural networks and variational inference. The key innovation was introducing a latent variable model with a well-defined probabilistic interpretation. VAEs enable efficient encoding of data and generation of new samples by sampling from the latent space. The training process of VAEs involves optimizing two objectives: the reconstruction loss, which ensures the faithful reconstruction of input data, and the Kullback–Leibler (KL) divergence, which encourages the latent space to follow a prior distribution, typically a multivariate Gaussian. Since their introduction, VAEs have gained significant attention and found numerous applications. They have been utilized in tasks such as image synthesis, anomaly detection, data generation, and representation learning. Researchers have explored various architectural modifications and training techniques to improve the quality of generated samples and address challenges like posterior collapse. The development of VAEs has opened up new possibilities in generative modeling and probabilistic inference, providing a versatile framework for learning and manipulating complex data distributions.
Leveraging the probabilistic approach integrated with an autoencoder helps VAEs to achieve great data generation performances compared to vanilla autoencoders (Kingma & Welling, 2019; Mayank Mittal & Harkirat Singh Behl, 2018). VAEs are also often compared with GANs in terms of their generation performances. The use of VAEs in SHM can go back to the early 2020s (Liu et al., 2019b; Ma et al., 2020), presenting anomaly detection on railways and feature extraction via VAE. Since then, several studies have been presented employing the generative skill of VAEs in civil SHM for various purposes, such as damage and anomaly identification, and condition assessment (Anaissi et al., 2023; Pollastro et al., 2022; Xu et al., 2021b; Yuan et al., 2021; Zhou et al., 2022), and optimal sensor placement (Sajedi & Liang, 2022) (Fig. 2), addressing data scarcity challenge in the SHM domain in one way or another.
2.2 Autoregressive models (AMs)
AMs have a history rooted in time series analysis and evolved with ML advancements. They gained prominence with autoregressive moving average (ARMA) models in the 1950s, which captured dependencies in time series data (Box, 1970). In the 1980s, AMs were applied to speech and audio processing, enabling the synthesis of realistic speech (Gray, 2010). The introduction of restricted Boltzmann machines in 2006 facilitated efficient training of autoregressive models. Notably, the development of deep AMs and PixelRNN (Oord et al., 2016a, 2016b) demonstrated the potential of DL in modeling pixel dependencies in images. The subsequent introduction of WaveNet (Oord et al., 2016a, 2016b) revolutionized AMs for speech and audio generation. Inspired by the transformer architecture (Vaswani et al., 2017), transformer-based autoregressive models further expanded AM capabilities of AMs across various domains. AMs continue to advance, leveraging DL techniques to model sequential dependencies and generate highly realistic and diverse samples.
AMs implicitly determine a distribution over sequences by using the chain rule for conditional probability. In this sequence, each step in the distribution is predicted based on the previous steps. Basically, AMs take the previous data in a sequence to predict a future value in that sequence. Thus, AMs are generally a better fit for time series with an intrinsic sequence of time steps, where they truly excel. One of the best-known models is WaveNet for audio generation (Oord et al., 2016a, 2016b). AMs are also used for images using sequential models for the pixels, such as the PixelRNN model (Oord et al., 2016a, 2016b) but are not great at image generation. Among other DGMs, it is essential to note that AMs are sequential but are still feedforward. Additionally, while they are generative, they still use a supervised approach. These facts make AMs faster, more stable in training (but very slow in data sampling and have poor scaling properties), and more straightforward and intuitive than the other DGMs. In civil SHM, AMs have been quite popular among researchers for years, and they were mainly used for feature extraction for damage identification using ARMA or variants (Entezami et al., 2021; Gul & Catbas, 2009; Liu et al., 2019a; Rajeev et al., 2022). AMs are also used for future data estimation (Psathas et al., 2022) (Fig. 3); however, feature extraction purposes have been seen more often in the literature. Figure 3 shows the estimated strain data from the train passage using WaveNet (Psathas et al., 2022).
2.3 Flow-based models (FBMs)
Flow-based models have emerged as a powerful class of generative models in recent years. The history of flow-based models can be traced back to the early 2010s (Dinh et al., 2014) when the concept of normalizing flows was introduced. Normalizing flows aim to model complex probability distributions by transforming simple distributions through a series of invertible mappings. In 2015, Dinh et al. proposed the Real NVP (real-valued non-volume preserving) architecture (Dinh et al., 2016), which allowed for flexible and tractable transformations in high-dimensional spaces. This marked a significant advancement in flow-based modeling. Subsequently, other flow-based architectures like Glow (Kingma & Dhariwal, 2018), FFJORD (Grathwohl et al., 2018), and Neural Spline Flows (Durkan et al., 2019) were introduced, further improving the expressiveness and scalability of flow models. FBMs have gained attention due to their ability to model complex data distributions, efficient sampling, and exact likelihood evaluation.
VAEs and GANs do not explicitly learn the probability density of real data, and they are intractable. FBMs (Danilo Jimenez Rezende & Shakir Mohamed, 2015) tackle this challenge by modeling a probability distribution using normalizing flows, a statistical tool for density estimation. In other words, FBMs learn the probability density explicitly, which makes them tractable. FBMs being tractable also makes the objective of the training simply the negative log-likelihood. Normalizing flows assist FBMs in modeling for a better distribution approximation leveraging the change-of-variable theorem of probabilities for transforming a distribution into a complex one. This is achieved by implementing a sequence of invertible transformation functions. The variables are repeatedly substituted for a new one based on the change-of-variable theorem to obtain a probability distribution of the end target variable. Essentially, FBMs are constructed by a sequence of invertible transformations with the aid of normalizing flows. Some notable FBMs are available in these references (Dinh et al., 2015; Kingma & Dhariwal, 2018). In addition, more recently, the normalizing flows were incorporated into a Diffusion Model (Qinsheng Zhang & Yongxin Chen, 2021). To the best of the authors' knowledge, the use of FBMs for SHM has not been observed during the preparation of this manuscript.
2.4 Energy-based models (EBMs)
Energy-based models (EBMs) have a long history in ML and have undergone several developments. The concept of EBMs can be traced back to the 1980s, and since then, the EBMs have been improved and extended (LeCun et al., 2006). When they were first introduced as a framework for unsupervised learning (Ackley et al., 1985; Hopfield, 1982), the Boltzmann machines were an early form of EBMs that employed the notion of energy to model joint probability distributions. However, training Boltzmann machines were computationally challenging. In recent years, the development of DL techniques and advancements in optimization algorithms have revitalized the interest in EBMs. Researchers have explored novel architectures, such as GANs and score-based models, to improve the learning and generation capabilities of EBMs. Ongoing research continues to refine and expand the applications of EBMs, making them promising tools for generative modeling, representation learning, and anomaly detection.
EBMs are a probabilistic model controlled by an energy function that defines the probability of a particular state. Essentially, they capture data dependencies by applying a probability scalar “energy” (a measure of compatibility) to each configuration of the variables. In that regard, inference includes setting the value of observed variables to 1 and then identifying the values of the rest of the variables that minimize that scalar energy amount. The learning can be accomplished by obtaining an energy function that correlates low energies with correct values of the rest of the variables and higher energies with incorrect values. EBMs use a unified framework combining all the probabilistic and non-probabilistic approaches for learning, especially for training graphical and structured models. The challenge of estimating normalization constant in probabilistic models does not exist in EBMs, which allows for more flexibility in the design of the learning process. However, EBMs suffer from modeling high-dimensional data. Although EBMs have been a research field for several decades, including some recent studies (Shuangfei Zhai & Cheng, 2016; Yilun Du & Igor Mordatch, 2019; Zhao et al., 2017), no studies are observed using EBMs in the civil SHM field, again to the best knowledge of the authors at this time.
2.5 Generative adversarial networks (GANs)
When GAN was first released in 2014 (Goodfellow et al., 2014), it received significant attention due to its novel approach (adversarial training concept – minimax game) and cutting-edge performance in image generation. GAN contains two networks: a generative network and a discriminator network. Essentially, the generator learns to generate similar data samples to the real dataset based on the discriminator’s output, while the discriminator also learns about the real data domain. In other words, both networks attempt to overcome each other in a minimax game; while the generator tries to fool the discriminator with the generated images, the discriminator tries to predict the synthetic and real images. Followed by its release, many researchers focused on improving the training of GAN due to its well-known unstable and no-convergence training process and mode collapse (less diversity in generated outputs) (Goodfellow, 2016; Salimans et al., 2016), WGAN (Arjovsky et al., 2017), WGAN-GP (Gulrajani et al., 2017). Moreover, there are many notable works using GANs, such as CycleGAN (Zhu et al., 2017), StyleGAN (Karras et al., 2018), and ESRGAN (Wang et al., 2018). Using GANs (original GAN and variants) for civil SHM applications is a popular research activity, and they were found beneficial for several challenges in SHM. For instance, lost data reconstruction (Fan et al., 2023; Jiang et al., 2022; Lei et al., 2021), data augmentation (Luleci et al., 2021), data domain translation (Luleci et al., 2023a, 2023b), anomaly and novelty detection (Soleimani‐Babakamali et al., 2022), and (Wang et al., 2019) data denoising. Figure 4 presents the use of GAN for true and reconstructed sensor data instances in their respective time and frequency domains. GANs were considered state-of-the-art generative models by many in terms of the quality of their generative performances until the recent rise of Diffusion Models (DMs).
2.6 Diffusion models (DMs)
The history of DMs can be traced back to 2015, inspired by non-equilibrium thermodynamics (Sohl-Dickstein et al., 2015), aiming to develop a learning approach that achieves analytical flexibility and tractability. Diffusion models, also known as denoising diffusion probabilistic models (DDPMs), have gained prominence in the field of generative modeling in recent years. These models employ a different approach compared to traditional generative models by explicitly modeling the process of iterative denoising a corrupted input to generate realistic samples.
The essential concept of DM is to successively add random noise to the data (image) through a Markov chain sequence to eventually obtain an isotropic Gaussian noise. Then, learn to reverse the forward diffusion process via backward propagation to reconstruct (or denoise) the desired data from the Gaussian noise. Some of the major differences of DMs between and the other DGMs are being able to generate highly realistic images and yield more diversity even better than GANs, having stable training procedures, and being able to be conditioned on a wide variety of inputs (Dhariwal & Nichol, 2021; Ho et al., 2020; Song & Ermon, 2019). One other unique property of DMs is that the latent space has the same dimensionality as the original data, which benefits DMs in terms of less computation. More recently, DMs have also shown remarkable success in the image and video generation, such as Imagen (Saharia et al., 2022) and Imagen Video (Ho et al., 2022) from Google, Dall-E 2 (Ramesh et al., 2022) from OpenAI and Make-A-Video (Singer et al., 2022) from Meta. Since DMs are a new research area in the Artificial Intelligence field, no study seems to have been in the literature using Diffusion Models in the SHM domain.
3 Discussion: DGMs and future directions in SHM
While each DGM has its drawbacks, some can be useful in civil SHM applications. VAEs enable efficient sampling from the latent space and can learn meaningful latent representations. However, they may produce blurry samples and struggle with capturing complex data distributions.
AMs are a good choice for time series-based applications, yet suffer from relatively slow inference and low QDS, unlike GANs. AMs can be computationally efficient but may struggle with capturing complex dependencies and generating high-dimensional data.
FBMs are much easier to converge and more stable during training, unlike VAEs and GANs. They provide exact likelihood estimation, enabling efficient density estimation and sampling. However, they can be computationally expensive during training and may struggle with modeling complex distributions.
EBMs show great out-of-distribution generalization skills thanks to their penalization learning strategy via the scalar energy values, making them a good candidate for knowledge transfer applications between dissimilar civil structures (Luleci & Catbas, 2022). EBMs can model complex data distributions and handle missing data, but they can be challenging to train and require sophisticated techniques for efficient inference.
Through adversarial training, GANs learn to generate high-quality and diverse samples. They have shown remarkable results in generating realistic samples but can be challenging to train and suffer from mode collapse, making training convergence difficult.
DMs reach state-of-the-art data generation performance; training them is easier and more stable, they are more explainable, and can be a better fit for time series due to their chain sequence approach. DMs provide a tractable likelihood estimation, making them suitable for density estimation tasks. They have shown promising results in generating high-quality samples but can be computationally expensive in sampling due to the iterative nature of the diffusion process.
Overall, these generative models have different strengths and limitations. The choice of model to use depends on the specific task, data characteristics, and trade-offs between sample quality, training stability, and computational efficiency, as presented in Table 1. From the previous paragraphs and Table 1, one can deduce that DMs should be the top-choice generative models. While this might be true, each DGM has its strengths and limitations. For instance, although DMs perform the best generative skills, the sampling speed is relatively slow compared to the other DGMs, forcing researchers to improve this limitation (Ulhaq et al., 2022). On the other hand, some other researchers combine the best of each DGM, such as training GANs with Diffusion (Wang et al., 2022), where the sampling speed is significantly improved. Another example is taking advantage of the strengths of normalizing flows and diffusion (Qinsheng Zhang & Yongxin Chen, 2021) (diffusion normalizing flow) to improve the training and sampling speeds of FBMs and DMs while enhancing the generation quality.
In general, DGMs could provide distinct ways to generate data for different goals, as mentioned in the introduction, such as:
-
Data generation (only) to generate data for general needs;
-
Lost data reconstruction to recover the lost or missing data due to SHM sensorial or transmission errors;
-
Data augmentation to improve the low performance in damage identification applications due to class imbalance of the training dataset;
-
Data domain translation to enable access to the paired data points for the latter damage identification applications;
-
Anomaly and novelty detection to identify anomalies, novelties, and outliers in structural response measurements, which could indicate potential issues with the structure;
-
Data denoising, deblurring, and repair to remove noise, blur, and enhance the quality of data;
-
Others, such as damage identification, annotation reduction via transfer learning to reduce data labeling for classification applications, or generating sensor placement layouts.
Future research directions for using DGMs in SHM could include several ways in which some are already being explored (Fig. 5).
Transfer learning and domain adaptation: Developing techniques for transferring knowledge and models from one structure to another or adapting pre-trained models to new structures. This includes adapting pre-trained models to new structures, leveraging domain knowledge, or simply building novel models for efficient generalization to other domains. This would facilitate the deployment of DGMs in real-world SHM scenarios where labeled data may be limited or unavailable, addressing the data scarcity issue.
Real-time monitoring and edge computing: Investigating the implementation of DGMs on edge devices or within the infrastructure of the monitored structures. This would enable real-time monitoring, analysis, and decision-making, reducing the dependence on cloud-based processing and enhancing the scalability and efficiency of SHM systems. For instance, in the event of lost data during monitoring, DGMs could assist in reconstructing the missing part in real time.
Integration of physics-based constraints: Investigating approaches incorporating physics-based constraints and structural mechanics principles into DGMs. This includes developing models that learn from data and capture the underlying physical behavior of structures, improving the accuracy and reliability of the SHM data or other SHM model generations.
Uncertainty quantification: Investigating methods to incorporate uncertainty estimation in DGMs for SHM. This includes developing probabilistic models that can provide confidence intervals or probability distributions for anomaly detection and damage assessment tasks, e.g., generating probability distributions for certain uncertainty ranges for different operational scenarios and structures.
Multi-modal data fusion: Exploring approaches to fuse data from multiple sensors and modalities using DGMs. This involves integrating different types of sensor data, such as vibration, strain, and temperature, to improve anomaly detection and provide a comprehensive understanding of structural health. For instance, a DGM could be used to generate temperature- and humidity-induced vibration data, providing a more holistic view of the operational status of the structure.
Human-in-the-loop interaction: Exploring interactive approaches that involve human experts in the loop to guide and refine the generative models' outputs. This would leverage the expertise and domain knowledge of human operators to enhance interpretability and reliability and supports decision-making processes in SHM applications.
DGMs also could be trained on structural response datasets for loading conditions like wind, earthquakes, or floods. These models could then be used to generate structural responses for different scenarios by the end-user for producing varying structural behavior simulations of how a structure is likely to respond to different types of loading, allowing engineers to understand the behavior of the structure better and identify potential areas of weakness. This could help engineers prioritize repairs and maintenance as well as take preventative measures to avoid catastrophic failures.
These future research directions aim to advance the use of DGMs in SHM by addressing key challenges, improving model performance, and facilitating their practical deployment in real-world structural monitoring scenarios.
4 Conclusion
The research and development in the civil SHM domain have been very progressive for the last few decades due to the increasing use of ML to tackle the challenging problems faced in the field (Avci et al., 2021; Azimi et al., 2020; Bao & Li, 2021). On the other hand, using deep generative models (DGMs) has also been a trend across many disciplines lately, demonstrating very efficient solutions for particular applications. Civil SHM is one of these disciplines that researchers have just begun exploring to use some members of DGMs towards SHM applications.
It is important to note that data scarcity is a significant challenge in civil SHM due to data collection tasks from civil structures being challenging. While data collection from every civil structure is not economically feasible, a large portion of the structures is worth monitoring due to the growing concern for the better management, operation, and safety of civil structures. Even when a few are monitored, SHM system-based (sensor or transmission errors) are typical, resulting in sensorial data loss. The fact that SHM applications, such as damage diagnosis and prognosis, rely on data-driven solutions makes the challenge of data scarcity even more significant. Therefore, employing deep generative models (DGMs) for SHM applications is critical, considering their excellent data generation performances as demonstrated in the literature.
While no studies exist using FBMs, EBMs, and DMs based on the literature review as of December 2022, quite a few works are available using GANs in the civil SHM domain. On the other hand, AMs are primarily used for feature extraction for damage identification, future data estimation, and similar applications. Lastly, several studies use VAEs in civil SHM for various purposes, such as anomaly detection, damage identification, and optimal sensor placement.
It can be argued that there is a large room for research and development using DGMs for data generative-based applications in the civil SHM field, particularly with case studies.
Availability of data and materials
Not applicable.
References
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive Science, 9(1), 147–169.
Anaissi, A., Zandavi, S. M., Suleiman, B., et al. (2023). Multi-objective variational autoencoder: An application for smart infrastructure maintenance. Applied Intelligence, 53, 12047–12062. https://doi.org/10.1007/s10489-022-04163-2
Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein GAN. Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research 70:214–223.
Avci, O., Abdeljaber, O., Kiranyaz, S., et al. (2021). A review of vibration-based damage detection in civil structures: From traditional methods to machine learning and deep learning applications. Mechanical Systems and Signal Processing, 147, 107077. https://doi.org/10.1016/j.ymssp.2020.107077
Azimi, M., Eslamlou, A., & Pekcan, G. (2020). Data-driven structural health monitoring and damage detection through deep learning: state-of-the-art review. Sensors, 20, 2778. https://doi.org/10.3390/s20102778
Bao, Y., & Li, H. (2021). Machine learning paradigm for structural health monitoring. Structural Health Monitoring, 20, 1353–1372. https://doi.org/10.1177/1475921720972416
Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022). Deep generative modelling: A comparative review of VAEs, GANs, normalizing flows, energy-based and autoregressive models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 7327–7347. https://doi.org/10.1109/TPAMI.2021.3116668
Box, G. E. P. (1970). Time series analysis; Forecasting and control. Holden-Day.
Catbas, F. N., Kijewski-Correa, T., & Aktan, A. E. (2013). Structural identification of constructed systems. American Society of Civil Engineers. https://doi.org/10.1061/9780784411971
Catbas, F. N., Luleci, F., Zakaria, M., et al. (2022). Extended reality (XR) for condition assessment of civil engineering structures: A literature review. Sensors, 22, 9560. https://doi.org/10.3390/s22239560
Chahal, K.S., He, M., Gao, A. et al. (2020). Energy-based models. https://atcold.github.io/pytorch-Deep-Learning/
Dhariwal, P., & Nichol, A. (2021). Diffusion models beat GANs on image synthesis. https://arxiv.org/abs/2105.05233
Dinh, L., Krueger, D., & Bengio, Y. (2014). NICE: Non-linear independent components estimation. International Conference on Learning Representations. https://arxiv.org/abs/1410.8516
Dinh, L., Krueger, D., & Bengio, Y. (2015). NICE: Non-linear independent components estimation. In: ICLR 2015.
Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016). Density estimation using Real NVP. Published as a conference paper at ICLR 2017.
Du, Y., & Mordatch, I. (2019). Implicit generation and modeling with energy-based models. In: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
Durkan, C., Bekasov, A., Murray, I., & Papamakarios, G. (2019). Neural spline flows. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
Entezami, A., Sarmadi, H., Salar, M., et al. (2021). A novel data-driven method for structural health monitoring under ambient vibration and high-dimensional features by robust multidimensional scaling. Structural Health Monitoring. https://doi.org/10.1177/1475921720973953
Fan, G., He, Z., & Li, J. (2023). Structural dynamic response reconstruction using self-attention enhanced generative adversarial networks. Engineering Structures, 276, 115334. https://doi.org/10.1016/j.engstruct.2022.115334
Goodfellow, I. (2016). NIPS 2016 Tutorial: Generative adversarial networks. http://arxiv.org/abs/1701.00160
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al. (2014). Generative adversarial networks. Proceedings of the 27th International Conference on Neural Information Processing Systems (Vol 2, pp 2672–2680). https://dl.acm.org/doi/https://doi.org/10.5555/2969033.2969125
Grathwohl, W., Chen, R.T.Q., Bettencourt, J., et al. (2018). FFJORD: Free-form continuous dynamics for scalable reversible generative models. Published as a conference paper at ICLR 2019.
Gray, R. M. (2010). Linear predictive coding and the internet protocol: A survey of LPC and a history of realtime digital speech on packet networks. Foundations and Trends.
Gul, M., & Catbas, F. N. (2009). Statistical pattern recognition for Structural Health Monitoring using time series modeling: Theory and experimental verifications. Mechanical Systems and Signal Processing, 23, 2192–2204. https://doi.org/10.1016/j.ymssp.2009.02.013
Gulrajani, I., Ahmed, F., Arjovsky, M., et al. (2017). Improved training of Wasserstein GANs. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17) (pp. 5769–5779). Curran Associates Inc., Red Hook, NY, USA.
Ho, J., Chan, W., Saharia, C., et al. (2022). Imagen video: High definition video generation with diffusion models. https://doi.org/10.48550/arXiv.2210.02303
Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.
Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8), 2554–2558.
Jiang, H., Wan, C., Yang, K., et al. (2022). Continuous missing data imputation with incomplete dataset by generative adversarial networks–based unsupervised learning for long-term bridge health monitoring. Structural Health Monitoring, 21, 1093–1109. https://doi.org/10.1177/14759217211021942
Karras, T., Laine, S., & Aila, T. (2021). A style-based generator architecture for generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(Dec. 2021), 4217–4228. https://doi.org/10.1109/TPAMI.2020.2970919
Kingma, D.P., & Dhariwal. P. (2018). Glow: Generative flow with invertible 1x1 convolutions. 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
Kingma, D.P., & Welling, M. (2013). Auto-encoding variational Bayes. https://arxiv.org/abs/1312.6114
Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders. Foundations and Trends in Machine Learning., 12(4), 307–392. https://doi.org/10.1561/2200000056
LeCun, Y., Chopra, S., & Hadsell, R. (2006). A tutorial on energy-based learning. In: Predicting Structured Data. MIT Press.
Lei, X., Sun, L., & Xia, Y. (2021). Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks. Structural Health Monitoring, 20, 2069–2087. https://doi.org/10.1177/1475921720959226
Liu, A., Wang, L., Bornn, L., & Farrar, C. (2019a). Robust structural health monitoring under environmental and operational uncertainty with switching state-space autoregressive models. Structural Health Monitoring, 18, 435–453. https://doi.org/10.1177/1475921718757721
Liu, J., Wei, Y., Bergés, M., et al. (2019b). Detecting anomalies in longitudinal elevation of track geometry using train dynamic responses via a variational autoencoder. In K.-W. Wang, H. Sohn, H. Huang, & J. P. Lynch (Eds.), Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems (p. 49). SPIE.
Luleci, F., Avci, O., & Catbas, F. N. (2023a). Improved undamaged-to-damaged acceleration response translation for structural health monitoring. Engineering Applications of Artificial Intelligence, 122, 106146. https://doi.org/10.1016/j.engappai.2023.106146
Luleci, F., & Catbas, F.N. (2022). Structural state translation: Condition transfer between civil structures using domain-generalization for structural health monitoring. https://doi.org/10.48550/arXiv.2212.14048
Luleci, F., & Catbas, F.N. (2023). Condition transfer between prestressed bridges using structural state translation for structural health monitoring. AI in Civil Engineering. https://doi.org/10.1007/s43503-023-00016-0
Luleci, F., Catbas, F. N., & Avci, O. (2021). Generative adversarial networks for labeled acceleration data augmentation for structural damage detection. J Civ Struct Health Monit. https://doi.org/10.1007/s13349-022-00627-8
Luleci, F., Catbas, F. N., & Avci, O. (2022). A literature review: Generative adversarial networks for civil structural health monitoring. Front Built Environ. https://doi.org/10.3389/fbuil.2022.1027379
Luleci, F., Catbas, F. N., & Avci, O. (2023b). CycleGAN for undamaged-to-damaged domain translation for structural health monitoring and damage detection. Mechanical Systems Signal Processing. https://doi.org/10.1016/j.ymssp.2023.110370
Luleci, F., Catbas, F. N., & Avci, O. (2023c). Generative adversarial networks for labeled acceleration data augmentation for structural damage detection. Journal of Civil Structural Health Monitoring, 13, 181–198. https://doi.org/10.1007/s13349-022-00627-8
Ma, X., Lin, Y., Nie, Z., & Ma, H. (2020). Structural damage identification based on unsupervised feature-extraction via variational auto-encoder. Measurement, 160, 107811. https://doi.org/10.1016/j.measurement.2020.107811
Malekzadeh, M., Atia, G., & Catbas, F. N. (2015). Performance-based structural health monitoring through an innovative hybrid data interpretation framework. Journal of Civil Structural Health Monitoring, 5, 287–305. https://doi.org/10.1007/s13349-015-0118-7
Mittal, M., & Behl, H.S. (2018). Variational autoencoders: A brief survey. https://mayankm96.github.io/assets/documents/projects/cs698-report.pdf
Pollastro, A., Testa, G., Bilotta, A., & Prevete, R. (2022). Semi-supervised detection of structural damage using variational autoencoder and a one-class support vector machine. IEEE, https://doi.org/10.1109/ACCESS.2023.3291674
Psathas, A.P., Iliadis, L., Achillopoulou, D., et al. (2022). Autoregressive deep learning models for bridge strain prediction (pp 150–164).
Rajeev, A., Pamwani, L., Ojha, S., & Shelke, A. (2022). Adaptive autoregressive modelling based structural health monitoring of RC beam-column joint subjected to shock loading. Structural Health Monitoring. https://doi.org/10.1177/14759217221101325
Ramesh, A., Dhariwal, P., Nichol, A., et al. (2022). Hierarchical text-conditional image generation with CLIP latents. https://doi.org/10.48550/arXiv.2204.06125
Rezende, D. J., & Mohamed, S. (2015). Variational inference with normalizing flows. In: Proceedings of the 32nd International Conference on Machine Learning. JMLR: W&CP vol 37, Lille, France.
Ruthotto, L., & Haber, E. (2021). An introduction to deep generative modeling. GAMM-Mitteilungen. https://doi.org/10.1002/gamm.202100008
Saharia, C., Chan, W., Saxena, S., et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding. https://doi.org/10.48550/arXiv.2205.11487
Sajedi, S., & Liang, X. (2022). Deep generative Bayesian optimization for sensor placement in structural health monitoring. Computer-Aided Civil and Infrastructure Engineering, 37, 1109–1127. https://doi.org/10.1111/mice.12799
Salimans, T., Goodfellow, I., Zaremba, W., et al. (2016). Improved techniques for training GANs. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS'16) (pp 234–2242). Curran Associates Inc., Red Hook, NY, USA.
Singer, U., Polyak, A., Hayes, T., et al. (2022). Make-a-video: Text-to-video generation without text-video data. Published as a conference paper at ICLR 2023. https://arxiv.org/abs/2209.14792
Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., & Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015 JMLR: W&CP volume 37.
Soleimani-Babakamali, M. H., Sepasdar, R., Nasrollahzadeh, K., et al. (2022). Toward a general unsupervised novelty detection framework in structural health monitoring. Computer-Aided Civil and Infrastructure Engineering, 37, 1128–1145. https://doi.org/10.1111/mice.12812
Soleimani-Babakamali, M. H., & ZakerEsteghamati, M. (2022). Estimating seismic demand models of a building inventory from nonlinear static analysis using deep learning methods. Engineering Structures, 266, 114576. https://doi.org/10.1016/j.engstruct.2022.114576
Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.
Tomczak, J. M. (2022). Deep Generative Modeling (1st ed.). Springer. https://doi.org/10.1007/978-3-030-93158-2
Ulhaq, A., Akhtar, N., & Pogrebna, G. (2022). Efficient diffusion models for vision: A survey. https://arxiv.org/abs/2210.09292
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. https://arxiv.org/abs/1706.03762?context=cs
van den Oord, A., Dieleman, S., Zen, H., et al. (2016b). WaveNet: A generative model for raw audio. https://arxiv.org/abs/1609.03499
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016a). Pixel recurrent neural networks. In: Proceedings of the 33rd International Conference on Machine Learning. JMLR.org
Wang, K., Zhang, X., Hao, Q., et al. (2019). Application of improved least-square generative adversarial networks for rail crack detection by AE technique. Neurocomputing, 332, 236–248. https://doi.org/10.1016/j.neucom.2018.12.057
Wang, X., Yu, K., Wu, S., et al. (2018). ESRGAN: Enhanced super-resolution generative adversarial networks. In: L. Leal-Taixé, S. Roth (Eds), Computer Vision—ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science (vol 11133). Springer, Cham. https://doi.org/10.1007/978-3-030-11021-5_5
Wang, Z., Zheng, H., He, P., et al. (2022). Diffusion-GAN: Training GANs with diffusion. https://arxiv.org/abs/2206.02262
Weng L. (2021). What are diffusion models? Lil’Log. In: https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.
Xu, Y., Lu, X., Cetiner, B., & Taciroglu, E. (2021a). Real-time regional seismic damage assessment framework based on long short-term memory neural network. Computer-Aided Civil and Infrastructure Engineering, 36, 504–521. https://doi.org/10.1111/mice.12628
Xu, Y., Tian, Y., Zhang, Y., & Li, H. (2021b). Deep-learning-based bridge condition assessment by probability density distribution reconstruction of girder vertical deflection and cable tension using unsupervised image transformation model. pp 35–45.
Yuan, Z., Zhu, S., Chang, C., et al. (2021). An unsupervised method based on convolutional variational auto-encoder and anomaly detection algorithms for light rail squat localization. Construction and Building Materials, 313, 125563. https://doi.org/10.1016/j.conbuildmat.2021.125563
Zhai, S., Cheng, Y., & Lu, W. (2016). Deep structured energy based models for anomaly detection. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48, New York
Zhang, Q., & Chen, Y. (2021). Diffusion s. In: 35th Conference on Neural Information Processing Systems.
Zhao, J., Mathieu, M., & LeCun, Y. (2017). Energy-based generative adversarial networks. In: Published as a conference paper at ICLR 2017.
Zhou, Y., Shu, X., Bao, T., et al. (2022). Dam safety assessment through data-level anomaly detection and information fusion. Structural Health Monitoring. https://doi.org/10.1177/14759217221117478
Zhu, J.-Y., Park, T., Isola, P., & Efros, A.A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. IEEE International Conference on Computer Vision (ICCV), Venice, Italy (pp. 2242–2251). doi: https://doi.org/10.1109/ICCV.2017.244
Acknowledgements
The research presented here is carried out at the Civil Infrastructure Technologies for Resilience and Safety (CITRS) Lab of the University of Central Florida. The authors particularly acknowledge Mahta Zakaria, Abdulrrahman AlGadi, and Inad Dkillord K. AlQurashi, who are CITRS members for language editing, proofreading, and other research discussions. The research was supported by NASA, and the authors greatly appreciate the agency’s support. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agency.
Funding
This study was supported by the National Aeronautics and Space Administration (NASA) Award No. 80NSSC20K0326 for the research activities and particularly for this paper.
Author information
Authors and Affiliations
Contributions
Conceptualization, F.L. and F.N.C.; methodology, F.L.; validation, F.L.; investigation, F.L. and F.N.C.; resources, F.N.C.; writing—original draft preparation, F.L.; writing—review and editing, F.L. and F.N.C.; supervision, F.N.C.; project administration, F.N.C.; funding acquisition, F.N.C. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no conflicts of interest in preparing this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Luleci, F., Catbas, F.N. A brief introductory review to deep generative models for civil structural health monitoring. AI Civ. Eng. 2, 9 (2023). https://doi.org/10.1007/s43503-023-00017-z
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s43503-023-00017-z