A deep multitask learning approach for air quality prediction

Abstract

Air pollution is one of the most serious threats to human health and is an issue causing growing public concern. Air quality forecasts play a fundamental role in providing decision-making support for environmental governance and emergency management, and there is an imperative need for more accurate forecasts. In this paper, we propose a novel spatial–temporal deep multitask learning (ST-DMTL) framework for air quality forecasting based on dynamic spatial panels of multiple data sources. Specifically, we develop a prediction model by combining multitask learning techniques with recurrent neural network (RNN) models and perform empirical analyses to evaluate the utility of each facet of the proposed framework based on a real-world dataset that contains 451,509 air quality records that were generated on an hourly basis from January 2013 to September 2017 in China. An application check is also conducted to verify the practical value of our proposed ST-DMTL framework. Our empirical results indicate the efficacy of the framework as a viable approach for air quality forecasts.

Appendix

1.1 Backpropagation through time (BPTT)

The basic equations of the LSTM applied in our paper are as follows (first introduced in Sect. 4.2):

$$ \begin{aligned} & Forecast_{t} = f_{F} \left( {H_{t} } \right) = W_{F} H_{t} + b_{F} \\ & H_{t} = G_{o, t} \odot \tanh C_{t} \\ & C_{t} = G_{f,t} \odot C_{t - 1} + G_{i,t} \odot \tilde{C}_{t} \\ & \tilde{C}_{t} = \tanh \left( {M_{C} X_{t} + N_{C} H_{t - 1} } \right) \\ \end{aligned} $$

The formulas for the three gates of input, forget and output are as follows:

$$ \begin{aligned} & G_{i, t} = \sigma \left( {M_{i} X_{t} + N_{i} H_{t - 1} + P_{i} C_{t - 1} } \right) \\ & G_{f, t} = \sigma \left( {M_{f} X_{t} + N_{f} H_{t - 1} + P_{f} C_{t - 1} } \right) \\ & G_{o, t} = \sigma \left( {M_{o} X_{t} + N_{o} H_{t - 1} + P_{o} C_{t} } \right) \\ \end{aligned} $$

The loss is defined as MSE, given by:

$$ L_{{MSE}} = Forecast_{t} - TrueValue_{{t2}}^{2} $$

Each training sample contains observations acquired during the time window considered in the temporal analysis, which can be regarded as a sequence whose length is $T$. As a result, the total error of the sample at time $t$ is the accumulated error from the observation at ${\text{t}} - T$ to that at $T$.

We calculate the gradient of our loss function with respect to each parameter in the model. Similar to the summation operation in loss, the gradient is summed at each time step from ${\text{t}} - T$ to $T$ for each training example.

The chain rule of differentiation is used in the gradient calculation process. We first take the partial derivative with respect to $Forecast_{t}$ and obtain:

$$ \delta Forecast_{t} = \frac{{\partial L_{MSE} }}{{\partial Forecast_{t} }} = 2\left( {Forecast_{t} - TrueValue_{t} } \right) $$

Then, we take the partial derivatives with respect to $W_{F}$,$H_{t}$,and $b_{F}$:

$$ \begin{aligned} & \delta W_{F} = \frac{{\partial L_{MSE} }}{{\partial W_{F} }} = \frac{{\partial L_{MSE} }}{{\partial Forecast_{t} }}\frac{{\partial Forecast_{t} }}{{\partial W_{F} }} = \delta Forecast_{t} H_{t} \\ & \delta b_{F} = \frac{{\partial L_{MSE} }}{{\partial b_{F} }} = \frac{{\partial L_{MSE} }}{{\partial Forecast_{t} }}\frac{{\partial Forecast_{t} }}{{\partial b_{F} }} = \delta Forecast_{t} \\ & \delta H_{t} = \frac{{\partial L_{MSE} }}{{\partial H_{t} }} = \frac{{\partial L_{MSE} }}{{\partial Forecast_{t} }}\frac{{\partial Forecast_{t} }}{{\partial H_{t} }} = \delta Forecast_{t} W_{F} \\ \end{aligned} $$

As $H_{t}$ depends on $G_{o, t}$ and $C_{t}$, we apply the chain rule again and obtain

$$ \begin{aligned} & \delta G_{o, t} = \frac{{\partial L_{MSE} }}{{\partial G_{o, t} }} = \frac{{\partial L_{MSE} }}{{\partial H_{t} }}\frac{{\partial H_{t} }}{{\partial G_{o, t} }} = \delta H_{t} \tanh C_{t} \\ & \delta C_{t} = \frac{{\partial L_{MSE} }}{{\partial C_{t} }} = \frac{{\partial L_{MSE} }}{{\partial H_{t} }}\frac{{\partial H_{t} }}{{\partial G_{o, t} }} = \delta H_{t} G_{o, t} \left( {1 - \tanh^{2} \left( {C_{t} } \right)} \right) \\ \end{aligned} $$

Additionally, $G_{o, t}$ depends on $M_{o}$,$N_{o}$ and $P_{o}$, so we obtain $\delta M_{o}$,$\delta N_{o}$ and $\delta P_{o}$ from the following:

$$ \begin{aligned} & \delta M_{o} = \frac{{\partial L_{MSE} }}{{\partial M_{o} }} = \frac{{\partial L_{MSE} }}{{\partial G_{o, t} }}\frac{{\partial G_{o, t} }}{{\partial M_{o} }} = \delta G_{o, t} \left( {1 - G_{o, t} } \right)G_{o, t} X_{t} \\ & \delta N_{o} = \frac{{\partial L_{MSE} }}{{\partial N_{o} }} = \frac{{\partial L_{MSE} }}{{\partial G_{o, t} }}\frac{{\partial G_{o, t} }}{{\partial N_{o} }} = \delta G_{o, t} \left( {1 - G_{o, t} } \right)G_{o, t} H_{t - 1} \\ & \delta P_{o} = \frac{{\partial L_{MSE} }}{{\partial P_{o} }} = \frac{{\partial L_{MSE} }}{{\partial G_{o, t} }}\frac{{\partial G_{o, t} }}{{\partial P_{o} }} = \delta G_{o, t} \left( {1 - G_{o, t} } \right)G_{o, t} C_{t} \\ \end{aligned} $$

For $C_{t}$, partial derivatives with respect to $G_{i, t}$, $G_{f, t}$ and $\tilde{C}_{t}$ are needed:

$$ \begin{aligned} & \delta G_{i, t} = \frac{{\partial L_{MSE} }}{{\partial G_{i, t} }} = \frac{{\partial L_{MSE} }}{{\partial C_{t} }}\frac{{\partial C_{t} }}{{\partial G_{i, t} }} = \delta C_{t} C_{t - 1} \\ & \delta G_{f, t} = \frac{{\partial L_{MSE} }}{{\partial G_{f, t} }} = \frac{{\partial L_{MSE} }}{{\partial C_{t} }}\frac{{\partial C_{t} }}{{\partial G_{f, t} }} = \delta C_{t} \tilde{C}_{t} \\ & \delta \tilde{C}_{t} = \frac{{\partial L_{MSE} }}{{\partial \tilde{C}_{t} }} = \frac{{\partial L_{MSE} }}{{\partial C_{t} }}\frac{{\partial C_{t} }}{{\partial \tilde{C}_{t} }} = \delta C_{t} G_{i, t} \\ \end{aligned} $$

Then, the gradients of the loss with respect to parameters $M$, $N$ and $P$ can be obtained:

$$ \begin{aligned} & \delta M_{S} = \frac{{\partial L_{MSE} }}{{\partial M_{S} }} = \frac{{\partial L_{MSE} }}{{\partial G_{S, t} }}\frac{{\partial G_{S, t} }}{{\partial M_{S} }} = \delta G_{S, t} \left( {1 - G_{S, t} } \right)G_{S, t} X_{t} \\ & \delta N_{S} = \frac{{\partial L_{MSE} }}{{\partial N_{S} }} = \frac{{\partial L_{MSE} }}{{\partial G_{S, t} }}\frac{{\partial G_{S, t} }}{{\partial N_{S} }} = \delta G_{S, t} \left( {1 - G_{S, t} } \right)G_{S, t} H_{t - 1} \\ & \delta P_{S} = \frac{{\partial L_{MSE} }}{{\partial P_{S} }} = \frac{{\partial L_{MSE} }}{{\partial G_{S, t} }}\frac{{\partial G_{S, t} }}{{\partial P_{S} }} = \delta G_{S, t} \left( {1 - G_{S, t} } \right)G_{S, t} C_{t} \\ \end{aligned} $$

where $S$ denotes which type of gate the parameter effects: $S \in \left\{ {i, f} \right\}$.Finally, the partial differential equations for $M_{C}$ and $N_{C}$ are as follows:

$$ \begin{aligned} & \delta M_{C} = \frac{{\partial L_{MSE} }}{{\partial M_{C} }} = \frac{{\partial L_{MSE} }}{{\partial \tilde{C}_{t} }}\frac{{\partial \tilde{C}_{t} }}{{\partial M_{C} }} = \delta \tilde{C}_{t} \left( {1 - \tilde{C}_{t}^{2} } \right)X_{t} \\ & \delta N_{C} = \frac{{\partial L_{MSE} }}{{\partial N_{C} }} = \frac{{\partial L_{MSE} }}{{\partial \tilde{C}_{t} }}\frac{{\partial \tilde{C}_{t} }}{{\partial N_{C} }} = \delta \tilde{C}_{t} \left( {1 - \tilde{C}_{t}^{2} } \right)H_{t - 1} \\ \end{aligned} $$

At this point, the gradients of the loss with respect to all parameters ($M_{i}$, $N_{i}$, $P_{i}$, $M_{f}$,$N_{f}$,$P_{f}$,$M_{o} N_{o}$,$P_{o}$ and $M_{C}$,$N_{C}$,$W_{F}$,$b_{F}$) have been obtained, and mini-batch gradient descent (MBGD) is used to learn the parameters (according to formula (22) in Sect. 4.2). Notably, our gradients depend only on the current values of terms on the right-hand side in the equations.