DAerosol-NTM: applying deep learning and neural Turing machine in aerosol prediction

Abstract

The pollution caused by aerosol (particulate matter) has a detrimental impact on urban environments, particularly in terms of socio-economic factors and public health. Aerosol particles, ranging in size from 1 nm to 100 µm, can easily penetrate organic tissues, carrying toxic gaseous compounds and minerals such as carbon monoxide, ozone, nitrogen dioxide, and sulfur dioxide. Recent advancements in neural network technology, combined with deep learning techniques, have made it possible to predict surges in aerosol pollution. In this study, we introduce DAerosol-NTM, a deep learning framework that utilizes the latest developments in neural Turing machines (NTMs) to access external memory. When compared with four baseline studies that employ multilayer perceptron (MLP), deep neural networks (DNNs), long short-term memory (LSTM), and deep LSTM (DLSTM), DAerosol-NTM significantly improves prediction accuracy by 8–31% and precision by 46–91% and reduces the root mean square error (RMSE) by 24–85%. Additionally, DAerosol-NTM incorporates up to 20 years of particulate matter data in its external storage, making it the first model capable of predicting aerosol pollution surges. By analyzing the data from the previous 96 h, the optimal time interval before and after the aerosol event (TIBAAE) enables the prediction of aerosol events within the following 24 h.

Appendix A: research concepts

This section addresses the theoretical and technical nuances and specialized terms used in the study and the advancements of DAerosol-NTM, including the AQI, deep learning (DL), deep short-term neural networks (DSTNN), and the neural Turing machine. For direct remarks on the materials and methods and the results of this paper, please visit the segments following “Research Concepts” section.

1.1 AQI index

AQI is an air quality indicator that reflects and evaluates the air quality status. Although the AQI scale is continuous, different descriptive categories have been implemented to ease public communication, as given in Table

Table 25 Different levels (AQI) [54]

25 [3, 54].

One crucial factor to note about AQI is its calculation method. The concentrations are independently measured using six reported pollutant parameters (SO2, NO2, PM2.5, PM10, O3, and CO). Any pollutant's highest value is taken as the AQI value [4]. For this reason, the calculation of AQI is driven by a single parameter, and the accurate prediction of all six pollutants is essential. In most urban areas, PM2.5 and PM10 are the leading drivers of AQI calculations and the dominating reason behind pollution and erosion.

1.2 Deep learning

Deep learning is machine learning, but it functions more similarly to the human brain in a deeper and more advanced form. In other words, deep learning is part of a more prominent family of machine learning that focuses on methods based on artificial neural networks. This type of learning is an essential element in data science that receives raw inputs and extracts high-level features in several layers, including statistics and forecasting modeling. In-depth learning is very beneficial for data scientists to collect, analyze, and interpret large amounts of data. In general, it makes the process faster and easier. In other words, deep learning models process data more accurately and quicker due to the complexity and high ability to learn, especially in big data in research fields such as image processing, pattern recognition, and computer vision [55]. Deep learning is a powerful machine learning method that provides approximation, classification, and predictability [57,58,59] and [56].

See Fig. 

Fig. 17

The difference between machine learning and deep learning 6 and [56]

17.

1.3 Deep short-term neural network

The LSTM architecture [55] works better than conventional neural networks for long-term tasks [58], including a deep and short-term neural network. Thus, the lower the short-term neural network output sequence, the higher the short-term neural network input sequence [61]. The deep LSTM architecture used in Refs. [7, 63] can be described by ($\sigma (x)$, $h_{t}^{l}$, $i_{t}^{l}$, $f_{t}^{l}$, $s_{t}^{l}$, $o_{t}^{l}$), where $l$ represents the layer index followed by Eqs. 4–9. Equation 10 defines the Softmax activity function for a simple N-dimensional problem.

$$\sigma (x) = \frac{1}{1 + \exp ( - x)}$$

(4)

$$h_{t}^{l} = o_{t}^{l} \tanh (S_{t}^{l} )$$

(5)

$$i_{t}^{l} = \sigma (W_{i}^{l} [X_{t} ;h_{t - 1}^{l} ;h_{t}^{l - 1} ] + b_{i}^{l} )$$

(6)

$$f_{t}^{l} = \sigma (W_{f}^{l} [X_{t} ;h_{t - 1}^{l} ;h_{t}^{l - 1} ] + b_{f}^{l} )$$

(7)

$$s_{t}^{l} = f_{t}^{l} s_{t - 1}^{l} + i_{t}^{l} \tanh (W_{s}^{l} [X_{t} ;h_{t - 1}^{l} ;h_{t}^{l - 1} ] + b_{s}^{l} )$$

(8)

$$o_{t}^{l} = \sigma (W_{o}^{l} [X_{t} ;h_{t - 1}^{l} ;h_{t}^{l - 1} ] + b_{o}^{l} )$$

(9)

$$S_{N} = \left\{ {a \in R^{N} :a_{i} \in [0,1],\sum \sum\nolimits_{{}}^{N} {a_{i} } = 1} \right\}$$

(10)

1.4 CNN-DLSTM

The use of classical CNN architecture is the best choice when input networks are 2-D or 3-D tensors like images or videos [64]. Since LSTMs architectures were more adapted for 1-D data, a new variant of LSTM called convolutional LSTM or ConvLSTM [65] is designed. In this architecture, the LSTM cell, which contains a convolution operation and input dimension of data, is kept in the output layer instead of just a 1-D vector. A convolution operation has replaced matrix multiplication at each gate of classical LSTM. The ConvLSTM architecture applies the capabilities of CNN and LSTM neural networks. It is normally developed for 2-D spatiotemporal data such as satellite images. In the first part of this model, convolutional layers extract essential features of the input data, and the results were flattened in a 1-D tensor so that they can use as input for the second part of the model (LSTM). Finally, before passing data in the last hidden layer, information has to be reshaped in the original form of input data. The architecture of CNN-LSTM is shown in Fig. 

Fig. 18

Architecture of the CNN-LSTM model [27]

18.

1.5 Neural Turing machine

It is a method derived from the Turing machine and neural networks. This model consists of recurrent neural networks (RNNs) [60] with an addressable external memory along with the ability of the recursive neural networks to perform algorithmic tasks such as sorting, copying, and N-gram. Generally speaking, a memory bank, a controller, and read and write heads are the main components of this method. The controller's job is to receive data from the outside and generate outputs during the update cycle. In addition, the neural Turing machine method guides the read and write heads directly into the external memory in the form of a tape [57, 66]. Figure 

Fig. 19

Structure of the neural Turing machine A in general and B in part [10] and [62]

19 shows the structure of the neural Turing machine in general in section (A) and its components in section (B) [10] and [62].

1.5.1 The actions that the controller performs through the heads

The controller consists of five main functions: read, write, erase, move to the next memory cell, and move to the previous one. During each update cycle, the network controller receives inputs from the external environment and publishes the outputs in response. The network also starts reading and writing from a memory matrix using parallel read and write heads. The dotted line in Fig. 19A shows the division between the NTM circuit and the outside world. Each part of the structure is recognizable and distinct. This feature makes the network easier to train with gradient descent by defining “blurry or unclear” reading and writing operations. It also communicates more or less with all memory elements (instead of considering one element as a typical Turing machine or digital computer). A “focus” mechanism determines the degree of blur.

Each operation forces the read/write heads to communicate with the small memory and ignore the rest. Because the interaction with memory is low and fragmented, NTM is based on data storage without interference. The output of the heads determines which memory location gets the most attention. These outputs define a normalized weighting on the rows of the memory matrix (referring to memory locations). At each assigned weight, each read/write head establishes the degree to which the head reads/writes in each area. Each head focuses precisely on a single memory location or multiple memory locations. Figure 

Fig. 20

How to access external memory [66]

20 displays the conceptual model of the controller's actions through the heads.

1.5.2 Reading and writing

The read and write operations are normalized weighting functions over the memory locations, similar to attention mechanisms. These weightings define a continuous distribution over the memory locations to make the operation differentiable. The reading operation is a simple linear combination of the memory locations:

Figure 

Fig. 21

Read and write operation in NTM [66]

21 shows the read and write operation in NTM [66] in sections (A) and (B).

Figure 21A shows the Nth element of W_t(i) according to the limits of Eq. 11:

$$\sum\limits_{i = 0} {W_{t} (i)} = 1 \to 0 \le W_{t} (i) \le 1,\forall i$$

(11)

M_t is the contents of the M*N memory matrix at time t, where N is the number of memory locations and M is the size of the vector in each location. W is the vector of weights on the N locations propagated by a head reading at time t since all weights are normalized.

The read vector is a weighted convex combination of the memory location. The length M reads the return r_t vector by the head defined as a combination of row vectors M_t(i) in memory, as shown in Eq. 12, separable in terms of memory and weight.

$$r_{t} \xleftarrow{{}}\sum\limits_{{}} {w_{i} } (i)M_{t} (i)$$

(12)

Figure 21B shows that the writing operation is a convex combination of erasing and writing to the memory locations. The write head outputs both erase (e) and add (a) vectors. Writing to the memory will then be made by erasing the locations defined by the write weighting vector and adding the locations specified by the same weighting vector. Again, notice that erasing and writing locations in different proportions make the operation differentiable, where parts of memory are erased according to the weighting vector, as shown in Eq. 13:

$$M^{\prime }_{t} (i) \leftarrow M_{t - 1(i)} [1 - w_{t} (i)e_{{t_{t} }} ]$$

(13)

Multiplying it in the memory location works as a peer-to-peer. Therefore, the elements of the memory location are zeroed where the weight is and the clearing element is 1. If the weight or clearance element is zero, the memory remains unchanged. When there are multiple writing heads, cleansing can be done in any order, where new information is added to locations defined by the weightings, as shown in Eq. 14:

$$M\mathop {_{t} }\limits^{{}} (i) \leftarrow M^{\prime}_{t} (i) + W_{t} (i)a_{t}$$

(14)

The combine, clear, and add operations produce the final memory content at time t. Because delete and add are different, compound writing operations are also distinguishable. Note that both addition and subtraction vectors have M-independent components. Again, adding the sequence of vectors by multiple heads is also trivial. It allows precise control over the modified elements in each memory location.

1.5.3 Addressing mechanism

In the previous section, the equations of reading and writing were shown and examined. However, no explanation was given on how the weights are produced. These weights are created by combining two addressing mechanisms with complementary features. The first mechanism is content-based addressing, which looks at locations based on the similarity between current values and values published by the controller that addresses the content of Hopfield networks. The advantage of content-based addressing is that the retrieval is easy. It only needs a controller to estimate a portion of the stored data to compare it to memory for the accurate stored value. However, content-based addressing is not suitable for solving all problems. In some works, the content of a variable is as desired [57]. However, the variable still needs a recognizable name or address. Computational problems are as follows: Variables X and Y can take both values, but the $f(x,y) = x \times y$ trend must still be defined. A controller initializes the x and y variables, stores them in different addresses, retrieves them, and performs a multiplication algorithm. In this case, the variables are addressable by location, not content. This form of addressing is called location-based addressing. Content-based addressing is more commonly used than location-based addressing, as the content of a memory location can contain location information within it. However, it is necessary to provide location-based addressing as a primary operation for some generalized forms in experiments. As a result, both mechanisms are used together. The flow diagram of the addressing mechanism indicates the sequence of operations to construct a weight vector while reading and writing [11] (see Fig.

Fig. 22

Flow diagram of the addressing mechanism [10] and [62]

22).

1.5.4 Network controller

The NTM structure is described in the section. It has several free parameters: memory size, number of read/write heads, and allowable location changes. Perhaps, the most important structure is the type of neural network used as the controller, especially when deciding whether to use a recurrent or a feedforward neural network (FNN). A recursive controller such as LSTM has internal memory to complete a larger matrix. Suppose a central processing unit controller on a digital computer (albeit with adaptive instructions instead of predefined ones) is compared to the RAM matrix. In that case, the hidden activities of the recursive controller are similar to the processor's registers. They allow the controller to combine information throughout the various time stages of the operation.

On the other hand, a feedforward controller mimics an RNN network by reading and writing to the specific memory location at each step. In addition, the feedforward controllers often give more transparency to network operations, because the read and write patterns in the memory matrix are usually easier to interpret than the internal state of an RNN. However, one of the limitations of a feedforward controller is that the number of read/write heads simultaneously causes a constraint (limiting operation) on the type of computation that NTM can perform. With a single read head, only a single conversion can be performed on a single memory vector at each time step. With two reading heads, it can do binary conversions, etc. Recurrent controllers can store reading vectors from previous efforts, so they do not suffer from this limitation.