Speech enhancement using U-nets with wide-context units

Abstract

In this article a new neural network for speech enhancement is proposed where single-channel noisy speech is processed in order to improve its intelligibility and quality. It is based on the U-net architecture, i.e. it is composed of two main blocks: encoder and decoder. Some of the corresponding layers in the encoder and decoder are connected with skip connections. In most of the encoder-decoder neural networks for speech enhancement known from the literature, the time-frequency resolution of the hidden feature maps is reduced. The main strategy in the presented approach is to maintain the time-frequency resolution of feature maps at all levels of the network while having large receptive field at the same time. In order to obtain features dependent on wide context we propose neural network units based on recurrent cells or dilated convolutions. The proposed neural network was evaluated using WSJ0 and TIMIT speech data mixed with noises from Noisex, DCASE and field recordings from Freesound online database. The results showed improvement over the baseline networks based on gated dilated convolutions or long-short term memory (LSTM) in terms of scale-independent speech-to-distortion ratio (SI-SDR), spectro-temporal objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) measures.

Appendix: Gated recurrent unit

There are many implementations and parameters of GRUs which can affect performance of the neural network. Therefore, in this appendix we provide information about variant of GRUs used in the experiments reported in this paper. The value of the reset gate is calculated as

$$ \textbf{r}_{t}=\sigma_{r}(\textbf{W}_{xr}\textbf{x}_{t}+\textbf{W}_{hr}\textbf{h}_{t-1}+\textbf{b}_{r}) , $$

(10)

where σ_r is the sigmoid, x_t is input vector to the GRU for time index t, while h_t− 1 is a vector representing previous state. Matrix W_xr, W_hr, and b_r are parameters of the reset gate. Similarly, update gate

$$ \textbf{u}_{t}=\sigma_{u}(\textbf{W}_{xu}\textbf{x}_{t}+\textbf{W}_{hu}\textbf{h}_{t-1}+\textbf{b}_{u}) , $$

(11)

and candidate state is computed using the formula

$$ \textbf{c}_{t}=\sigma_{c}(\textbf{W}_{xc}\textbf{x}_{t}+\textbf{r}_{t}\odot (\textbf{W}_{hc}\textbf{h}_{t-1})+\textbf{b}_{c}) , $$

(12)

where σ_c is tanh function, W_xc, W_hc and b_c are weights and bias. Finally, state h_t is computed using

$$ \textbf{h}_{t}=(\textbf{1}-\textbf{u}_{t})\odot \textbf{h}_{t-1} + \textbf{u}_{t} \odot \textbf{c}_{t} . $$

(13)

The processing performed by GRU can be described as a linear transformation

$$ \left[\begin{array}{cc} \textbf{W}_{xr} & \textbf{W}_{hr} \\ \textbf{W}_{xu} & \textbf{W}_{hu} \\ \textbf{W}_{xc} & \textbf{0} \\ \textbf{0} & \textbf{W}_{hc} \\ \textbf{0} & \textbf{I} \end{array}\right]\left[ \begin{array}{c} \textbf{x}_{t} \\ \textbf{h}_{t-1} \end{array}\right] + \left[ \begin{array}{c} \textbf{b}_{r} \\ \textbf{b}_{u} \\ \textbf{0} \\ \textbf{0} \\ \textbf{0} \end{array}\right] = \left[ \begin{array}{c} \textbf{z}_{r} \\ \textbf{z}_{u} \\ \textbf{z}_{xc} \\ \textbf{z}_{xh} \\ \textbf{h}_{t-1} \end{array}\right] , $$

(14)

which is transformed using the following nonlinear function

$$ f(\textbf{z}) = (1-\sigma(\textbf{z}_{u})\odot \textbf{h}_{t-1} + \sigma(\textbf{z}_{u})\odot \tanh(\textbf{z}_{xc}+\sigma(\textbf{z}_{r})\odot\textbf{z}_{hc}+\textbf{b}_{c}) . $$

(15)

Additionally, in all experiments featuring recurrent layers, the initial states of recurrences were also learned during training. We found this to have a small but consistent positive effect on the network’s performance.