1 Introduction

The changes of weather, environmental erosion, natural disasters and increasing traffic loadings can continually modify the behavior and even cause deterioration of bridges in their long-term service [1]. Application of BHM has been recognized as an attractive tool to improve the health and safety of bridge and provide early warning on structure damage [2,3,4]. A typical BHM system generally provides various effective real-time information such as temperature, wind, crack, deflection and stress. We can use those information to judge the bridge health status according to the corresponding diagnostic method. Thus, bridge health monitoring and safety assessment are inevitable requirements for the sustainable development of bridge engineering.

We know that the detection efficiency of sensors installed in the bridge construction gradually deteriorates during their long-term operation which affects the reliability of monitoring data. In addition, the severe environment conditions have a serious impact on the quality of the collected data [2]. What is more, the types of collected data are different and its amount is huge. Therefore, the difficulty for the current research and application of bridge health monitoring system is often not collecting the monitoring data, but how to extract useful information from the massive data accumulated over time [5]. However, most of bridge management systems currently use the data collected from the sensors without analysis which will greatly affect the quality of the data and lead the bridge structure damage identification and fault diagnosis technology to fall into a “data disaster” dilemma. The purpose of this paper is using methods based on data mining to overcome the problems of subjectivity, high computation complexity, low sensitivity and complicated technology in traditional analysis.

With the uninterrupted work of the bridge monitoring data acquisition system, a large amount of data are transmitted and stored in the database every day. Therefore, the accumulated data in the database can be said to be massive during the entire bridge operation period. Traditional database technology has shown its weakness in dealing with such a large amount of data storage and processing. The emergence of technology such as Hadoop provides an excellent solution to massive data processing.

Data processing is a core component of bridge health monitoring system. High-quality data are the basis for accurately judging the health status of bridges. With the increasing scale of bridge monitoring data, traditional database technology has gradually weakened in storing and processing data. In order to meet the challenges brought by the data growth in the bridge monitoring system, the bridge data monitoring and prediction function combined with Hadoop have theoretical significance and practical value. In this work, we built a triple-node Hadoop cluster and tested the proposed algorithm in a distributed environment.

In this paper, we focus on processing the monitoring data as well as getting structured data. And then based on the clustering theory of data mining, we use the Kohonen neural network to group the structured data and explain the point in clusters. We have also established a LSTM network model to predict the deflection which can reflect the health of the bridge structure by analyzing the historical monitoring data. These two methods both can be used to evaluate the health status of bridges to some degree. The former can get the classification pattern of the data collected under normal operating condition and then provide corresponding early warning through the analysis of outliers generated when new data were input to the network.

The latter can predict the value of a specific attribute, and then we can obtain the health status by comparing the predicted value with its limited value under normal operation. In brief, the Kohonen neural network can make a general assessment of the overall structure of the bridges, and LSTM neural network can predict the value of a specific bridge’s parameter. These two approaches have their own emphasis, but also both can evaluate the healthy status of bridges. The main contribution and work are as follows:

  1. 1.

    We preprocess the data collected by the sensors of bridge health monitoring system. Firstly, we fill the missing values by analyzing the range of data values as well as the historical data and empirical data. Secondly, we analyze the cause of abnormal data and repair them. Finally, we merge the data that are closest in acquisition time and then standardize each record.

  2. 2.

    We use Kohonen neural network to self-adaptively cluster the processed structured data and get the classification pattern of the data collected under normal operating condition. Then, we can analyze the outliers generated when new data were input to the network, and provide corresponding early warning according to the results.

  3. 3.

    We use the time series prediction model established by LSTM neural network to predict the deflection value. And then we can obtain the health status by comparing the predicted value with its limit value under normal operation.

2 Literature review

In order to ensure the safety and durability of bridges, the research and evaluation system for bridge health monitoring has been developed in the twentieth century and has made extensive achievements [6,7,8]. The UK first developed a long-term health monitoring devices and data acquisition processing system in the early 1980s [9]. This research finding was applied to continuous steel box girder Foyle Bridge in Northern Ireland. This system is the earliest, relatively complete structural monitoring system. In the mid-later 1980s, the USA began to install sensors on a number of bridges, such as the Sunshine Sky Bridge in Florida on which there were more than 500 sensors installed to measure the temperature, strain and displacement.

BHM in China began in the 1990s, when different scales monitoring systems were installed with the construction of a large number of bridges [10]. In 2000, the Hong Kong Highways Department installed a large number of sensors for temperature, wind, strain, displacement and acceleration monitoring on Tsing Ma Bridge and Tingkau Bridge. As far as the Tsing Ma Bridge is concerned, more than 800 functional sensors were installed including GPS surveying instrument and anemometer. And the Nanjing Yangtze River Bridge, Xupu Bridge and Humen Bridge were all equipped with the monitoring system to monitor the structural response shortly after the project completed.

Hadoop, an open-source distributed framework launched by Apache in 2007, is a big data solution widely promoted and applied by academia and industry. Hadoop mainly includes Hadoop Distributed File System (HDFS) and Map Reduce, a parallel computing model. Hadoop is widely used for large-scale data processing on clusters. In terms of productivity and maturity, it is better to run them on a supercomputer than to develop a new framework from scratch. Many supercomputing centers have begun to officially support the Hadoop environment [11].

The easy operation of environmental vibration test makes it a widely used method in bridge structure detection, but it can only obtain the basic modal parameters such as frequency, mode and dump ratio [12, 13]. Often, such methods provide a global analysis of the structure integrity, but won’t give the location of the damage which leads to ineffective decision-making for structural management. In order to overcome the limitation and solve the challenge of damage detection, many authors have proposed to use the data mining approach for health assessment in BHM. Researchers have been using time series and chaotic theory to predict the information of deflection at mid-span Ma Sangxi Bridge [14]. They found it is effective in predicting chaotic time series by using multi-step recurrent BP neural network and RBF neural network. The results show that multi-recursive BP neural network can predict the health of bridges. Another study proposes a clustering method to group the nodes with similar behavior on bridges and then detects the abnormal joints [15]. This method learns the most representative behavioral models from historical data and detects the potential damage in the structure with distance obtained from the normal model. All in all, data mining technology has great advantages in dealing with the massive data collected by the bridge health monitoring system.

Time series analysis is the study of data collected in chronological order. In general, time series contain data sequences obtained within a fixed sampling time. At present, time series prediction is widely used. A practical time series prediction-based non-stationarity detection method was proposed [16]. Chen et al. [17] introduced a new wind speed prediction method based on LSTMs, SVRM and EO deep learning time series prediction nonlinear and learning integration. Based on the nonlinear learning integration of LSTM, SVRM and EO, the proposed LSTM system achieved satisfactory wind speed prediction performance. In [18], machine learning algorithm was used to predict the wind speed of Osorio wind farm. The optimal intelligence model was introduced at different time intervals. The past data were the input, and the future data were used to represent the MLA output. Additionally, in [19], the author proposed a new multi-attention-based network to predict geographic sensory time series based on heterogeneous data in multiple fields.

3 Method

In bridge health monitoring, it is unclear what kind of relationship exists in a large amount of historical data. The classification results will not be very good due to the lack of classification rules in advance. Therefore, it is more appropriate to use clustering method to deal with health monitoring data because it is not necessary to understand attribute relationship in data. In addition, the bridge structure damage is gradually formed in the course of its operation and we should excavate the law of structure performance changing with the time from a large number of data samples. Therefore, a long short-term memory neural network is used to predict the value of some important parameters which can reflect the bridge health state.

3.1 Kohonen neural network

The Kohonen network is a special neural network for self-organizing maps. We can use this network to divide the data into distinct categories when we don’t know which categories are included in the dataset. The Kohonen network is shown in Fig. 1. It has two layers: input layer and output layer (also known as the competition layer) [20]. The dimension of the input vector determines the number of neurons in the input layer, and the neurons of the output layer are arranged in a two-dimensional plane. There are weights among the neurons in the output layer which is different form the traditional BP network that only has weights among the neurons of the input layer and the neurons of the output layer. The training steps of the network are as follows:

Fig. 1
figure 1

Kohonen neural network structure

Full size image

  • Step 1: Randomly initializing all weights \(W_{ij}(i=1,2,\dots m,\, m\) is the total number of input nodes, \(j=1,2,\dots ,n\), n is the total of number output nodes.)

  • Step 2: Computing the Euclidean distance \(||X_i - W_j||\) between each input sample and the weight vectors, where \(X_i\) representing the \(i_{\mathrm{th}}\) sample.

  • Step 3: Defining the neuron C with the smallest Euclidean distance \(X_i - W_j = {\mathrm{min}}||X_j - W_j||\) as the winner. And we denote the winner’s neighborhood as \(N_c(t)\).

  • Step 4: Adapting the weights of the winner and specified neighbors.

    $$\begin{aligned} \left\{ \begin{array}{ll} W_{ij}(t+1) = W_{ij}(t) + \alpha (t)[W_{ij}(t) - X_i], &{}\quad j\in N_c(t)\\ W_{ij}(t) = W_{ij}(t),&{}\quad j\notin N_c(t) \end{array} \right. \end{aligned}$$
    (1)

    where \(W_{ij}(t)\) represents the weights between neurons i and neurons j that within a neighborhood at time t and \(\alpha (t) = \alpha (0)(1 - t/T)\) is the learning rate. T is the whole time to train.

  • Step 5: Adjusting neighborhood size and learning rate. It is obvious that the network converges when the learning rate \(\alpha (t)\) decreases to nil or \(N_c(t)\) shrinks to an acceptable neighborhood range. Then we will record it.

  • Step 6: Training the network continuously in the aforementioned manner until its satisfies the conditions.

  • After sufficient training, the Kohonen network will provide a graph in which several independent regions are formed to represent specific clusters.

3.2 Long short-term memory neural network

The LSTM neural networks in our work are used for bridge health monitoring, which are originally used to deal with time series problem. The LSTM network is an improved structure of recurrent neural network (RNN). RNN is a feed forward network with feedback loops and internal memories [21]. They store sequences in their internal memory for a long time, which is helpful for making accurate decision of the next step compared with the traditional feed forward networks. In this paper, we adopt the LSTM framework proposed in [22] as is illustrated in Fig. 2.

Fig. 2
figure 2

LSTM neural network

Full size image

There are three layers of LSTM networks: input layer, LSTM layer and regression layer. \(X_t\) and \(Y_t\) are input data samples and output prediction results, respectively. In terms of bridge health monitoring, X represents extracted features that have an impact on the health of the bridge and Y the deflection data that can reflect the health state of the bridge. The LSTM layer uses a special hidden unit called memory cell as the core of LSTM neural network to memorize the information of long-term input sample. It solves the problem of vanishing gradient descent problem in the traditional RNN. As the core of an LSTM neural network, the LSTM layer uses a special hidden unit called memory cell to remember long-term input sample information which resolves the vanishing gradient descent problem in the traditional RNN. The memory unit is composed of three main parts: input gate, output gate and forget gate. We use \(h_t\) to denote the output of the memory cell which can be calculated through the following equations:

$$\begin{aligned} f_t = \sigma (W_{xf}X_t + W_{hf}h_{t-1}+b_f) \end{aligned}$$
(2)
$$\begin{aligned} i_t = \sigma (W_{xi}X_t + W_{hi}h_{t-1}+b_i) \end{aligned}$$
(3)
$$\begin{aligned} o_t = \sigma (W_{xo}X_t + W_{ho}h_{t-1}+b_o) \end{aligned}$$
(4)
$$\begin{aligned} C_t = f_t * C_{t-1} + i_t *\tanh (W_{xc}X_t + W_{hc}h_{t-1} + b_c) \end{aligned}$$
(5)
$$\begin{aligned} h_t = o_t * \tanh (C_t) \end{aligned}$$
(6)

where \(i_t\), \(o_t\), \(f_t\), and \(C_t\) are, respectively, the value of input gate, output gate, forget gate and cell state at each time t; \(W_{xf}\), \(W_{xi}\) and \(W_{xo}\) are weight matrices for the hidden state of the previous to the current; \(W_{hf}\), \(W_{hi}\) and \(W_{ho}\) are weights matrices for input to hidden state link; \(b_f\), \(b_i\), \(b_o\) and \(b_c\) are bias vectors; \(X_t\) is the current input; \(h_{t-1}\) is the LSTM layer’s output at the previous time \(t-1\); \(\sigma\) is the sigmoid activation function; and ‘*’ is the scalar multiplication.

Finally, we use the output \(h_t\) to predict \(Y_t\):

$$\begin{aligned} Y_t = \sigma (W_{hy}h_t + b_y) \end{aligned}$$
(7)

The output \(h_t\) of memory cell is updated step by step according to previous information. In order to minimize the usual square error given by function f, we train the neural network by using the backpropagation with gradient descent:

$$\begin{aligned} f=\sum _{t}||y_t - Y_t||^2 \end{aligned}$$
(8)

During the update process, the forget gate decides how much previous information should be discarded from the cell state; the input and output activations of memory cells are controlled by input gates and output gates, respectively. In addition, the model parameters of weights and bias are updated during training. As mentioned earlier, memory cells are composed of gates which are responsible for storing, reading or writing information to the network, so that the LSTM network can get bridge health status according to previous bridge monitoring information.

4 Discussion and experiments

The bridge monitoring system will operate 24 h after installation, and the monitoring data will be uploaded and stored in real time. Assume that the stored data in a day are 5G, and then we can know that the storage capacity of 1 year is about 1825G. It will not only be meaningless but also consume a lot of storage space if we store all these information but rarely use it. Therefore, in this chapter, we will analyze the data collected by the bridge monitoring system in order to mine useful information from the massive data.

Before the experiment, we built and configured the Hadoop distributed cluster. We used three nodes. We called the master node command lab01, and the other two child nodes were named lab02 and lab03. A root user was created on each computer.

4.1 Preprocessing bridge monitoring data

In this paper, we take the monitoring data from July 10, 2017, to July 9, 2018, of a bridge in Hubei Province as samples. The main monitoring parameters of this bridge include temperature, humidity, wind, deflection, strain and crack. The number of measuring points for each parameter is 40, 24, 6, 56, 108 and 8, respectively. In order to make more effective use of these data and improve the accuracy and efficiency of mining, it is very important to process data in advance. We should make corresponding measures to deal with the existing problems in the raw data collected by sensors:

  • In the health monitoring system, the initial time from collecting data of sensors with different attributes is different and the acquisition frequency is also different. Then merge the data, we combine the data that are closest in acquisition time, according to the highest frequency of items. The total number of samples is 8735 after merging.

  • In addition, data may be missing due to short-term severe environmental change or interference in the process of data transmission. The missing values can be replaced by the record of the previous time because the parameters of bridge won’t change very much in a short time.

  • Analyzing the variation trend of the local maximum and minimum value of each parameter can make us understand the variation characteristics of local time and control the change trend of bridge structure. As shown in Fig. 3, the maximum value of deflection1 has a tendency to decrease. In order to facilitate the future data analysis, we need to find the maximum, minimum and average value for each parameter. The statistics of some properties are shown in Table 1.

  • The different ranges of monitoring values from various types of sensors make it possible to neglect the properties with small values in modeling and training. Therefore, in order to improve the effectiveness of the model and balance the influence of each attribute on the model, it is necessary to standardize the values of each attribute. The standardization process is as follows:

    $$\begin{aligned} y_i = \frac{x_i - A_{{\mathrm{min}}}}{A_{{\mathrm{max}}}-A_{{\mathrm{min}}}} \end{aligned}$$
    (9)

    where \(x_i\) and \(y_i \,(i=1,2,3,\dots ,n)\) are the original and normalized value, respectively, and \(A_{{\mathrm{min}}}\) and \(A_{{\mathrm{max}}}\) represent the minimum and maximum value of attribute A, respectively. The standardized values of some attributes are shown in Table 2.

  • Principal components analysis (PCA) can create a replacement, smaller set of variables to combine the essence of the original attributes for compression and dimensionality reduction in bridge monitoring data. We can redefine each property and get new variables through PCA. These variables are independent of each other, which is beneficial for the precise training of the model. In this study, we use PAC to reduce the data to 200 dimensions to be the input of Kohonen clustering network because the clustering time is shorter and the score according to the clustering analysis result of the Calinski–Harabase index evaluation is higher in this case. Therefore, we use PCA to reduce the original data to 200 dimensions for better clustering results.

Fig. 3
figure 3

The maximum changing trend of deflection1

Full size image
Table 1 The statistical values of some properties
Full size table
Table 2 Some data records after standardization
Full size table

4.2 Clustering analysis model based on Kohonen neural network

In this paper, the preprocessed data are used as the input of the Kohonen network, and a \(3*2\) Kohonen network is established to analyze the bridge monitoring data (including deflection, temperature, humidity, strain, crack). The analysis of data by Kohonen is divided into two process: One is the general evaluation process which is used to obtain the overall pattern of the data. In this process, the initial learning rate \(\alpha\) is set to 0.3 and the neighboring \(N_c\) is set to 2; the other is the optimization process which is used to adjust the mapping network to build a better mining model. In the process, the learning rate \(\alpha\) is set to 0.1 and the neighboring \(N_c\) is set to 1. After many iterations, the clustering results are shown in Figs. 4 and 5. Figure 4 shows the result of 242-dimensional original data without performing PCA. We can see that the data in a cluster are scattered and the clustering results are not obvious. Figure 5 represents the result of using PCA to denoise data by reducing the dimension of the data. We focus on the analysis of Fig. 5. Figure 5 shows that the data are clustered into 4 groups. Each record belongs to the corresponding cluster, and one cluster variable is added to original data for analysis. The distance between clusters is shown in Table 3. We can know that the distance between clusters represents the relevance of data and with greater distances comes greater difference between clusters. And the number of data records in each generated cluster is shown in Table 4. The average and standard deviation of the corresponding data records in each cluster are shown in Tables 5 and 6.

Fig. 4
figure 4

Original data (242)

Full size image
Fig. 5
figure 5

PCA (200)

Full size image
Table 3 The analysis of clustering result about distance between clusters
Full size table
Table 4 The number of proportion in each cluster
Full size table
Table 5 The average value of some attributes in different clusters
Full size table
Table 6 The standard deviation of some attributes in different clusters
Full size table

We can also see from Fig. 5 that Cluster 1 and Cluster 2 have relatively large discreteness which is because there is a big difference between the record in Cluster 1 and Cluster 2. The temperature and deflection values are lower in Cluster 1. On the contrary, the temperature and deflection values are relatively high in Cluster 2. This coincides with the principle of clustering based on the similarity and dissimilarity in data which means that clustering can find out the data’s inner characteristic and distribution rule.

The clustering of the above Kohonen network forms 4 distinct clusters which generally contain patterns of monitoring data under normal condition, and each cluster represents a category with similar properties. Then, we can input the new data into the Kohonen neural network and compare the clustering results (including the proportion of each cluster, the average, standard deviation of each property) with the results obtained from the normal data. If there is a large deviation, the data may be abnormal. Therefore, the Kohonen clustering model can be used for outlier detection and provide an early warning reference for bridge structure.

4.3 Times series prediction based on LSTM

In the bridge structure, the characteristics of grider deflection (according to which we can make a timely response to the bridge structure condition under the joint action of load and external environment, because it is not prone to noise and is sensitive to structure damage) can truly reflect the bridge health condition. Therefore, we use the LSTM neural network to predict the deflection in BHM.

We use the architecture described in earlier to develop the LSTM work.Then we take the temperature, crack, humidity, strain and deflection as inputs to predict the value of deflection. The total number of input and output vectors is 8735. We take the first 8435 as training set and the remaining data as the prediction set. We train a 3-layer LSTM model and set the dimensionality of cells state and hidden state to be 512. In order to balance well the performance and training time, we set time step, batch size and learning rate to 3, 128, 0.00044, respectively. What is more, we will use the back propagation with gradient descent to train the weight metrics so as to minimize the mean square error (MSE). The MSE can be calculated as follows:

$$\begin{aligned} {\text{MSE}} = \frac{1}{n}\sum _{i=1}^{n}(y_i - Y_i)^2 \end{aligned}$$
(10)

where \(y_i\) and \(Y_i\) represent the real and predicted value of deflectioni. The model has converged after 40,000 iterations, and Fig. 6 shows some of the real value and prediction results. We can see that the difference between the predicted and real value is not huge. In order to better understand the prediction effect of the LSTM model, we calculate the MSE value of each deflection and the ratio of the predicted values that meet different accuracy requirements. The results are shown in Table 7.

Fig. 6
figure 6

The true versus predicted value of some deflections

Full size image
Table 7 MSE and the ratio of different accuracy for some deflections
Full size table

In terms of the difference between the predicted and real value for deflection25, 83% of the data are within 0.5 mm, 95% are within 1 mm and 99% are within 2 mm. We also can know that the predicted deflection value of the small MSE is close to the real deflection. We can identify the bridge health state through the predicted deflection value. Firstly, we take the maximum deflection obtained under the effect combination of serviceability limit state and bearing capacity limit state as threshold and then compare the predicted value with the threshold, Finally, we provide early warning if the predicted value goes above threshold.

5 Conclusion

With the continuous development of bridge construction as well as more and more bridges being put into operation, the amount of data generated in the bridge monitoring process is also increasing and the structure of data is also more and more complex. The problem that urgently needs to be solved is how to extract effective information from these data to guide the maintenance and operation of bridge.

In this paper, we analyze the data collected by sensors in the bridge monitoring system and give the method of data preprocessing based on the theory of data mining. We use Kohonen neural network to self-adaptively group the processed structured data and get the classification pattern of the data collected under normal operating condition which can provide corresponding early warning and find the abnormal changes of bridge structure through the analysis of outliers. In addition, we use the time series prediction model established by LSTM neural network to predict the deflection value. At the best predictive point, 99% of the data have an error of less than 0.5 mm, and 74% of the data in worst-performing points meet this accuracy requirement. What is more, all deflection points have at least 95% of their data showing a prediction error of less than 1 mm. Therefore, LSTM can be used to predict the value of monitoring parameters in BHM. We can observe the changing trend of bridge structure by comparing the predicted value with the limit value under normal operation. All work above is still in the initial stage of exploration. In future work, we will focus on transforming it from theory to engineering practice.