Glottal Activity Detection from the Speech Signal Using Multifractal Analysis

Abstract

This work proposes a novel method for the detection of glottal activity regions from the speech signal. Glottal activity detection refers to the problem of discriminating voiced and unvoiced segments of the speech signal. This is a fundamental step in the work flow of many speech processing applications. Much of the existing approaches for voiced/unvoiced detection are based on linear measures though the speech is produced from an underlying nonlinear process. The present work solves the problem from a nonlinear perspective, using the framework of multifractal analysis. The fractal property of the speech signal during the production of voiced and unvoiced sounds is sought to obtain the characterization of glottal activity. The characterization is done by computing the Hurst exponent from the evaluation of the scaling property of fluctuations present in the speech signal. Experimental analysis shows that Hurst exponent varies consistently with respect to the dynamics of glottal activity. The performance of the proposed method has been evaluated on the CMU-arctic, Keele and KED-Timit databases with simultaneous electroglottogram signals. Experimental results show that the average detection accuracy or error rate of the proposed method is comparable to the best performing algorithm on clean speech signals. Besides, evaluation of the robustness of the proposed method to noise degradation shows comparable results with other methods for signal-to-noise ratio greater than 10 dB and 20 dB, respectively, for white noise and babble noise.

Appendices

A Surrogate Test and Relevance of Fractal Analysis

Basically, the surrogate test examines the validity of a null hypothesis $({H_0})$ formulated for the original time series. ${H_0}$ is formulated based on the origin of the time series under investigation. Now, surrogates for the original time series are created by a surrogate algorithm, preserving the amplitude distribution, autocorrelation, local mean and variance of the original data. Then, a discriminating statistic is measured for the original data $({T_\mathrm{o}})$ and its surrogates $({T_\mathrm{s}})$. Also, the distribution of ${T_\mathrm{s}}$ obtained for the surrogate data can be estimated. If ${T_\mathrm{o}}$ and ${T_\mathrm{s}}$ are significantly different, the null hypothesis ${H_0}$ can be rejected. However, the rejection is carried out in a probabilistic manner, specifying the significance level. The significance level is determined by the number of surrogate data sets as, $p = 2 \times \frac{1}{{\left( {{{{n}}_\mathrm{s}} + 1} \right) }}$ where ${{{\text {n}}_\mathrm{s}}}$ is the total number of surrogates. The measure of significance for rejection is given by

$$\begin{aligned} t = \frac{{\left| {{T_\mathrm{o}} - \mathrm{mean}({T_\mathrm{s}})} \right| }}{{{\sigma _\mathrm{s}}}} \end{aligned}$$

(10)

where ${{\sigma _\mathrm{s}}}$ is the standard deviation of the distribution for surrogates.

The general techniques employed for the generation of surrogate data are random permutation (RP), Fourier transform (FT) and amplitude-adjusted Fourier transform (AAFT) [41, 48]. RP shuffles the data so that any linear correlations present in the original data may be destroyed. However, surrogates preserve the amplitude distribution of the original data. The FT surrogate is based on the null hypothesis that the data originate from a linear Gaussian process. It maintains the amplitude spectrum but the phases are randomized. AAFT algorithm was developed based on the null hypothesis that the original data are a monotonic nonlinear transformation of a linear Gaussian process. Here, the original data are initially rescaled to preserve the Gaussian property. Then, FT surrogates are generated and rescaled back to the amplitude distribution of the original data. Further, the choice of the discriminating statistic is also crucial in the surrogate analysis. The statistic should be selected in such a way that it should not preserve the hypothesis governing the surrogate algorithm. For example, the selection of mean as a discriminating statistic will always satisfy the null hypothesis governing random permutation. In this work, we use the HE estimated using MFDFA as the discriminating statistic.

We use an ensemble of 99 surrogates of the same length as the original speech signal (‘kdt001’ from KED-Timit) in each of the surrogate methods. Figure 11 shows the distribution of the discriminating statistics for the surrogates $({T_\mathrm{s}})$ and the original speech signal $({T_\mathrm{o}})$. The corresponding measure of significance t obtained for each surrogate test is also shown in Fig. 11. Here, ${H_0}$ for each surrogate method is rejected if the measure of significance (t) is greater than 2. This is evident from Fig. 11a where the null hypothesis governing RP surrogates is rejected, revealing some temporal correlation in the speech signal. Such a correlated time series is then tested using FT surrogates. If FT is also rejected, then the analysis is continued with AAFT surrogates. Here, we observe that ${H_0}$ governing FT and AAFT are also rejected (Fig. 11b, c). The rejection of ${H_0}$ governing RP, FT and AAFT is an indication of nonlinear process involved in the production of the speech signal. Further, we verified results of the surrogate test in two ways. Firstly, a voiced segment from the same utterance is simulated using LP coefficients and impulse train as excitation. The LP coefficients are obtained by LP analysis (of order 12) of the original speech segment. We use a frame size of 20 ms and frame shift of 10 ms for the LP analysis. The simulated speech output is tested using RP and FT surrogates. We find that the ${H_0}$ governing RP is rejected and FT is accepted (Fig. 12). Secondly, a synthetic version of the aforesaid Timit utterance is generated using ‘HTS-2005’ online tool [11], which uses a source–filter model. Again, we observe that ${H_0}$ governing FT is accepted (Fig. 13b). This is an indication of the linear process involved in the production of both the simulated (LP-based) and HTS synthetic speech signals. Thus, we conjecture that the source–filter model gives a linear approximation of the speech signal. The actual speech production process involves nonlinear interaction of the subsystems/parts from lungs to lips. In this context, a nonlinear technique like fractal analysis appears to be appropriate/relevant for the analysis of speech signals.

Fig. 11

Surrogate test for the speech signal ‘kdt001’ from KED-Timit database. Distribution of HE for the surrogates generated using a RP, b FT and c AAFT. The statistic for the original utterance is shown as a vertical dashed line (red). The measure of significance t is indicated on top right for each method (Color figure online)

Fig. 12

Surrogate test for the simulated speech signal using LP analysis and impulse excitation. Distribution of HE for the surrogates generated using a RP and b FT. The statistic for the original utterance is shown as a vertical dashed line (red). The measure of significance t is indicated on top right for each method (Color figure online)

Fig. 13

Surrogate test for the synthetic version of ‘kdt001’ speech signal generated using HTS-2005, which uses source–filter model. Distribution of HE for the surrogates generated using a RP and b FT. The statistic for the original utterance is shown as a vertical dashed line (red). The measure of significance t is indicated on top right for each method (Color figure online)

In order to investigate the relevance of fractal analysis in characterizing GA/non-GA regions of the speech signal, we have conducted the following experiment. Firstly, we took the aforesaid simulated voiced speech (/aa/) and its original version. Then, we perform MFDFA on these segments. The multifractal spectrum obtained for each of these segments is shown in Fig. 14c. It is observed that the multifractal spectrum of both the segments is clustered near to zero. Secondly, we simulated an unvoiced speech segment (/s/) from the same Timit utterance using LP analysis and random noise excitation. Then, we repeated the fractal analysis on the simulated unvoiced speech and its original version. Figure 15c depicts the resultant multifractal spectrums. From the plot, we can infer that the original unvoiced speech corresponds to a large inverted arc (indicating its strong multifractal nature). In contrast, the spectrum of the simulated unvoiced speech corresponds to a small arc (indicating an approximately monofractal nature). Precisely, the multifractal nature of the unvoiced speech is lost in the LP analysis. This is due to the limitation of the LP analysis in identifying the source–filter interaction. The LP analysis relies on the assumption that source and filter are decoupled entities and the excitation source has no influence on the vocal tract system parameters. However, the excitation source can influence the vocal tract system parameters in the actual speech production process. This in turn can contribute to spatial and temporal variations in the speech signal, resulting in its multifractal nature.

Fig. 14

Multifractal analysis on original and simulated voiced speech signal. a Original voiced speech signal, corresponding b simulated version, c multifractal spectrum

Fig. 15

Multifractal analysis on original and simulated unvoiced speech signal. a Original unvoiced speech signal, corresponding b simulated version, c multifractal spectrum

Thus, we can conclude that the simulated voiced speech preserves the fractal nature of the original version. However, the simulated unvoiced speech did not capture the spatial and temporal variations (multifractal nature) in the original version. Therefore, the characterization/detection of GA/non-GA (voiced/unvoiced) region based on multifractal analysis may not be effective in the simulated speech signal using LP coefficients and impulse/random noise excitation.

B A Comparative Study of Different Ground-Truth GA Estimation Methods

The ground-truth GA regions of the speech signal from CMU-arctic and KED-Timit databases can be estimated from the corresponding EGG signals. Here, we investigated the performance of three state-of-the-art methods, such as the ZFF method, the ’SIGMA’ algorithm and the VMD method in estimating the ground-truth GA regions from the EGG signal. The ZFF method uses a simple threshold operation on the SoE at the estimated epochs (from the EGG) for the identification of reference GA regions. Precisely, an epoch with SoE greater than one percent of the maximum SoE is considered as the ground-truth GA region [12]. Using the SIGMA algorithm, the boundaries of GA regions are estimated based on the initial and final instants of glottal closure [26]. That is, if the distance between consecutive GA regions is greater than twice the maximum pitch period, the corresponding region is marked as a non-GA region. The VMD method relies on the estimation of epochs from the mode which oscillates close to the fundamental frequency of the speech signal [27, 28]. For GA regions, the method estimates epochs as the positive zero crossings of the selected mode.

Fig. 16

Comparison of estimated ground-truth GA regions using different methods. a EGG signal with actual GA regions indicated using dashed blue line, b DEGG signal with estimated epochs from SIGMA marked using ‘$*$,’ c SoE obtained from the ZFF signal, d VMD output signal with estimated epochs marked using ‘$*$. The estimated GA regions are shown in thick red line on b–d (Color figure online)

Figure 16 shows the demonstration of GAD from the EGG signal (taken from the Keele database since it has proper ground truth) using the aforementioned methods. The detected GA/non-GA regions from these methods are marked using a thick line (red) on the corresponding output signals (Fig. 16b–d). Reference GA/non-GA regions are shown in Fig. 16a using a dashed blue line. From the visual inspection of Fig. 16c, d, it is clear that the VMD method and ZFF method outperform SIGMA in estimating GA regions, especially at voicing offset regions (region indicated using dashed circle). Therefore, one can choose either the ZFF method or the VMD method for estimation of ground-truth GA regions. Nevertheless, we perform a performance evaluation of the proposed method (and other state-of-the-art methods) using ground-truth GA regions obtained from the ZFF method and VMD method. The evaluation is done on a test database (here, we choose CMU-arctic) since a comprehensive study on ground-truth GAD algorithms is beyond the scope of the present work. Precisely, we compute the $VUV _\mathrm{E}$ during the detection of GA regions from the speech signal with respect to references from the ZFF method and VMD method. Figure 17a–c shows the $VUV _\mathrm{E}$ obtained for each method (with respect to the two references) on BDL, SLT and JMK databases, respectively. From the results, it is evident that the $VUV _\mathrm{E}$ is nearly equal for the two references. Hence, the choice of a method for the estimation of ground-truth GA region is not very critical.

Fig. 17

A comparative evaluation of the performance of the proposed method and state-of-the-art methods in GAD from speech signal using the ZFF- and VMD-based ground-truth references. Evaluation on a BDL, b SLT and c JMK databases. I—ZFF method, II—SRH method, III—GEFBA, IV—proposed method

C Phase Space Reconstruction

Generally, all variables defining the state of a complex system are not available in practice. In such a scenario, the dynamics of the system can be visualized by reconstructing the phase space of evolution of the measured variable or time series [33]. The reconstructed phase space will be topologically equivalent to the original one. Also, the variation in the dynamics is always found to be reflected in the structure of the phase space. Therefore, we employ the phase space reconstruction method to visualize the dynamics of the speech production system during the production of voiced and unvoiced speech sounds. We use the time delay embedding technique [46] for reconstruction. Precisely, the speech data are converted into an ensemble of delay vectors. Each delay vector corresponds to a state in the reconstructed phase space. For faithful reconstruction, the embedding parameters such as optimum time delay $\tau$ and least embedding dimension $d_0$ are properly determined. $\tau$ is obtained by computing the first minimum of the average mutual information (AMI). Further, the false nearest neighbors (FNN) technique is used for determining $d_0$.

For a demonstration of the difference in complexity of the phase space corresponding to the GA/non-GA, we took the same voiced and an unvoiced segment shown in Fig. 4. Then, optimum time delay ${\tau }$ and minimum embedding dimension ${d_0}$ are computed for the segments separately. Figure 18b, d shows the reconstructed phase space for the voiced (Fig. 18a) and unvoiced segments (Fig. 18c), respectively. By visual inspection of Fig. 18b, it is clear that the phase space for voiced segment shows circular patterns (closed trajectories) representing periodic/quasiperiodic oscillations. This occurs due to the periodic/quasiperiodic vibration of vocal folds during the production of voiced speech sounds. On the other hand, the attractor for unvoiced sound (Fig. 18d) is expanded in all directions indicating irregular fluctuations in the airflow. The reconstructed phase space also reveals that the voiced and unvoiced sounds possess different multifractal structure and can be characterized based on multifractal measures.

Fig. 18

Demonstration of the reconstructed phase space of the voiced and unvoiced segment of a speech signal. a, b A voiced speech segment and corresponding reconstructed phase space, c, d an unvoiced speech segment and corresponding reconstructed phase space

D Phoneme Category of Sounds

Table 7 gives the phoneme category of sounds mentioned in Figs. 3,4, 8 and  9.

Table 7 Phoneme category of sounds