Modeling State-Conditional Observation Distribution Using Weighted Stereo Samples for Factorial Speech Processing Models

Abstract

This paper investigates the effectiveness of factorial speech processing models in noise-robust automatic speech recognition tasks. For this purpose, the paper proposes an idealistic approach for modeling state-conditional observation distribution of factorial models based on weighted stereo samples. This approach is an extension to previous single-pass retraining for ideal model compensation which is extended here to support multiple audio sources. Non-stationary noises can be considered as one of these audio sources with multiple states. Experiments of this paper over the set A of the Aurora 2 dataset show that recognition performance can be improved by this consideration. The improvement is significant in low signal-to-noise energy conditions, up to 4 % absolute word recognition accuracy. In addition to the power of the proposed method in accurate representation of state-conditional observation distribution, it has an important advantage over previous methods by providing the opportunity to independently select feature spaces for both source and corrupted features. This opens a new window for seeking better feature spaces appropriate for noisy speech, independent from clean speech features.

Appendix: Extending the EM Algorithm for Modeling Mixture of Gaussians Based on Weighted Samples

In the E-step of the EM algorithm for weighted particles, particle weights have no effect on the component responsibility equations. By considering particle weights as the replicating order of the particles (similar to (24)), we see that this replication has no effect on the component responsibilities to each particle. Therefore, component responsibilities are calculated without considering particle weights by the old parameter set as in E-step of the standard EM algorithm for GMMs:

$$\begin{aligned} \gamma _{l} \left( k \right) \propto \pi _{k}^{\prime } \mathcal {N}\left( {{\varvec{y}}_{{{l}}} ;\varvec{\mu }_{{{k}}}^{\prime } ,\varvec{\Sigma }_{{{k}}}^{\prime } } \right) \end{aligned}$$

(33)

where the normalization constant is $\mathop \sum \nolimits _{k=1}^K \pi _{k}^{\prime } \mathcal {N}\left( {\varvec{y}_{l} ;\varvec{\mu }_{k}^{\prime } , \varvec{\Sigma }_{k}^{\prime } } \right)$.

For the M-step, the following optimization problem must be solved:

$$\begin{aligned}&\theta ^\mathrm{new}=\mathop {\hbox {argmax}}\limits _\theta \mathcal {Q}\left( {\theta ,{\theta }^{'}} \right) \nonumber \\&\hbox {st}:\mathop \sum \nolimits _{k=1}^K \pi _{k} =1 \end{aligned}$$

(34)

Using the method of Lagrange multiplier for satisfying the constraint for component priors, we have the following objective function for optimization:

$$\begin{aligned}&g\left( {\varvec{\mu },\varvec{\Sigma },\varvec{\pi }} \right) \nonumber \\&\quad =\mathop \sum \nolimits _{l=1}^L w_{l} \mathop \sum \nolimits _{k=1}^K \gamma _{l} \left( k \right) \left[ {\ln p\left( {{\varvec{y}}_{{{l}}} ;\varvec{\mu }_{{{k}}} ,\varvec{\Sigma }_{{{k}}} } \right) +\ln \pi _{k} } \right] +\lambda \left( {\mathop \sum \nolimits _{k=1}^K \pi _{k} -1} \right) \nonumber \\ \end{aligned}$$

(35)

Taking the derivative g with respect to $\varvec{\mu }_{k}$ results in:

$$\begin{aligned} \partial g/\partial \varvec{\mu }_{{{k}}} =2\mathop \sum \nolimits _{l=1}^L w_{l} \gamma _{l} \left( k \right) \left[ {\varvec{\Sigma }_{{{k}}}^{-1} \left( {{\varvec{y}}_{{{l}}}-\varvec{\mu }_{{{k}}} } \right) } \right] \end{aligned}$$

(36)

Now (30) is easily obtained for updating $\varvec{\mu }_{{{k}}}$ by setting this derivative to zero. For estimating $\varvec{\Sigma }_{{{k}}}$, according to [6] the derivative takes the following form:

$$\begin{aligned} \partial g/\partial \varvec{\Sigma }_{{{k}}} =-\frac{1}{2}\mathop \sum \nolimits _{l=1}^L w_{l} \gamma _{l} \left( k \right) \left[ {\varvec{\Sigma }_{{{k}}}^{-1} - \varvec{\Sigma }_{{{k}}}^{-1} \left( {{\varvec{y}}_{{{l}}} -\varvec{\mu }_{{{k}}} } \right) \left( {{\varvec{y}}_{{{l}}} -\varvec{\mu }_{{{k}}} } \right) ^{T} \varvec{\Sigma }_{{\varvec{k}}}^{-1}} \right] \end{aligned}$$

(37)

in which the $\varvec{\mu }_{k}$ is estimated by (30). Setting it to zero, we obtain:

$$\begin{aligned} \mathop \sum \nolimits _{l=1}^L w_{l} \gamma _{l} \left( k \right) \left( {{\varvec{y}}_{{{l}}} -\varvec{\mu }_{{{k}}} } \right) \left( {{\varvec{y}}_{{{l}}} -\varvec{\mu }_{{{k}}} } \right) ^{T}\varvec{\Sigma }_{k}^{-1} =\mathop \sum \nolimits _{l=1}^L w_{l} \gamma _{l} \left( k \right) \end{aligned}$$

(38)

Then (31) is obtained for estimating $\varvec{\Sigma }_{{{k}}}$ in which when the number of samples are significant, there is no need for adjusting the estimator for bias. Finally for $\pi _{k}$ we have:

$$\begin{aligned} \partial g/\partial \pi _{k} =\mathop \sum \nolimits _{l=1}^L \left( {w_{l} \gamma _{l} \left( k \right) } \right) /\pi _{k} +\lambda =0 \end{aligned}$$

(39)

by using the assumption $\mathop \sum \nolimits _{k=1}^K \pi _{k} =1$ and considering $\gamma _{l} \left( k \right)$ as a valid conditional probability mass function, $\lambda$ is calculated by:

$$\begin{aligned} \lambda =-\mathop \sum \nolimits _{l=1}^L w_{l} \end{aligned}$$

(40)

Now we can eliminate $\lambda$ from (39) by (40) which leads to (32) for updating $\pi _{k} .$