NASR: NonAuditory Speech Recognition with Motion Sensors in Head-Mounted Displays
With the growing popularity of Virtual Reality (VR), people spend more and more time wearing Head-Mounted Display (HMD) for an immersive experience. HMD is physically attached on wearer’s head so that head motion can be tracked. We find it can also detect subtle movement of facial muscles which is strongly related to speech according to the mechanism of phonation. Inspired by this observation, we propose NonAuditory Speech Recognition (NASR). It uses motion sensor for recognizing spoken words. Different from most prior work of speech recognition using microphone to capture audio signal for analysis, NASR is resistant to acoustic noise of surroundings because of its nonauditory mechanism. Without using microphone, it consumes less power and requires no special permissions in most operating systems. Besides, NASR can be seamlessly integrated into existing speech recognition systems. Through extensive experiments, NASR can get up to 90.97% precision with 82.98% recall rate for speech recognition.
KeywordsHead-Mounted Display Motion sensor Speech recognition Machine learning
- 2.Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437 (2013)Google Scholar
- 3.Hernandez, J., Li, Y., Rehg, J.M., Picard, R.W.: Bioglass: physiological parameter estimation using a head-mounted wearable device. In: International Conference on Wireless Mobile Communication and Healthcare, pp. 55–58 (2014)Google Scholar
- 4.Hernandez, J., McDuff, D.J., Picard, R.W.: Biophone: physiology monitoring from peripheral smartphone motions. In: International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 7180–7183 (2015)Google Scholar
- 5.Hernandez, J., McDuff, D., Picard, R.W.: Biowatch: estimation of heart and breathing rates from wrist motions. In: Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare, pp. 169–176 (2015)Google Scholar
- 6.Kwon, S., Lee, J., Chung, G.S., Park, K.S.: Validation of heart rate extraction through an iPhone accelerometer. In: International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5260–5263 (2011)Google Scholar