Auxiliary Features from Laser-Doppler Vibrometer Sensor for Deep Neural Network Based Robust Speech Recognition
Recently, the signals captured from a laser Doppler vibrometer (LDV) sensor have shown the noise robustness to automatic speech recognition (ASR) systems by enhancing the acoustic signal prior to feature extraction. In this study, an alternative approach, namely concatenating the auxiliary features extracted from the LDV signal with the conventional acoustic features, is proposed to further improve ASR performance based on the deep neural network (DNN) for acoustic modeling. The preliminary experiments on a small set of stereo-data including both LDV and acoustic signals demonstrate its effectiveness. Thus, to leverage more existing large-scale speech databases, a regression DNN is designed to map acoustic features to LDV features, which is well trained from a stereo-data set with a limited size and then used to generate pseudo-LDV features from a massive speech data set for parallel training of an ASR system. Our experiments verify that both the features from the limited scale LDV data set as well as the massive scale pseudo-LDV features can yield significant improvements of recognition performance over the system using purely acoustic features, in both quiet and noisy environments.
KeywordsLaser Doppler vibrometer Auxiliary features Deep neural network Regression model Speech recognition
This work was supported in part by the National Natural Science Foundation of China under Grants 61671422 and U1613211, in part by the National Key Research and Development Program of China under Grant 2017YFB1002200, in part by the MOE-Microsoft Key Laboratory of USTC.
- 1.Baker, J.M., Deng, L., Glass, J., Khudanpur, S., Lee, C.H., Morgan, N., & O’Shaughnessy, D. (2009). IEEE Signal Processing Magazine, 26(3).Google Scholar
- 5.Hinton, G. (2010). Momentum, 9(1), 926.Google Scholar
- 6.Han, K., He, Y., Bagchi, D., Fosler-Lussier, E., & Wang, D. (2015). In Proceedings of Interspeech (pp. 2484–2488).Google Scholar
- 10.Du, J., Wang, Q., Gao, T., Xu, Y., Dai, L.R., & Lee, C.H. (2014). In Proceedings of interspeech (pp. 616–620).Google Scholar
- 11.Bagchi, D., Mandel, M.I., Wang, Z., He, Y., Plummer, A., & Fosler-Lussier, E. (2015). . In Proceedings IEEE ASRU.Google Scholar
- 13.Kinoshita, K., Delcroix, M., Yoshioka, T., Nakatani, T., Sehr, A., Kellermann, W., & Maas, R. (2013). In 2013 IEEE workshop on applications of signal processing to audio and acoustics (WASPAA) (pp. 1–4).Google Scholar
- 14.Vincent, E., Barker, J., Watanabe, S., Le Roux, J., Nesta, F., & Matassoni, M. (2013). In 2013 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 126– 130).Google Scholar
- 15.Harper, M. (2015). In 2015 IEEE workshop on automatic speech recognition and understanding (ASRU) (pp. 547–554).Google Scholar
- 16.Dekens, T., Verhelst, W., Capman, F., & Beaugendre, F. (2010). In 2010 18th European signal processing conference (pp. 1978–1982): IEEE.Google Scholar
- 17.Liu, Z., Zhang, Z., Acero, A., Droppo, J., & Huang, X. (2004). In 2004 IEEE 6th workshop on multimedia signal processing (pp. 363–366). IEEE.Google Scholar
- 18.Radha, N., Shahina, A., Vinoth, G., & Khan, A.N. (2014). In 2014 international conference on control instrumentation, communication and computational technologies (ICCICCT) (pp. 1343–1348). IEEE.Google Scholar
- 19.Breguet, J., Pellaux, J.P., & Gisin, N. (1994). In 10th optical fibre sensors conference (pp. 457–460). International Society for Optics and Photonics.Google Scholar
- 22.Goode, R.L., Ball, G., Nishihara, S., & Nakamura, K. (1996). Otology & Neurotology, 17(6), 813.Google Scholar
- 23.Avargel, Y., & Cohen, I. (2011). In 2011 joint workshop on hands-free speech communication and microphone arrays (HSCMA) (pp. 109–114). IEEE.Google Scholar
- 24.Avargel, Y., Bakish, T., Dekel, A., Horovitz, G., Kurtz, Y., & Moyal, A. (2011). In Proceedings speech process, conference. Israel: Tel-Aviv.Google Scholar
- 25.Xie, Z., Du, J., McLoughlin, I., Xu, Y., Ma, F., & Wang, H. (2016). In Proceedings of ISCSLP.Google Scholar
- 27.Seltzer, M.L., Yu, D., & Wang, Y. (2013). In 2013 IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 7398–7402). IEEE.Google Scholar
- 28.[online]. iflytek. http://www.iflytek.com/.
- 29.Gao, T., Du, J., Dai, L.R., & Lee, C.H. (2015). In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE.Google Scholar
- 30.[online]. vocalzoom. http://vocalzoom.com/.