Development of a Silent Speech Interface for Augmented Reality Applications

Abstract

Silent speech interfaces using non-invasive electromyography (EMG) sensors have been utilized to control internet-of-things devices [1] and provide communications in acoustically challenging environments [2]. However, they have yet to be implemented into Augmented Reality displays, an area they can potentially revolutionize as a human-machine interface by offering low-profile and fluid input. This study overviews the development of a silent speech interface that receives and decodes input from subvocalizations recorded by skin surface EMG sensors, to be used to control a heads-up-display built on a Microsoft HoloLens. Measured muscle activation of the anterior cervical region while a subject subvocalized words from a predetermined library were collected. Trials consisting of subvocalized words were parsed for individual subvocalizations to build a dataset for training of a speech recognition model. The speech recognition model based on a one dimensional convolutional neural network employed to classify subvocalized words was built with the Keras application programming interface in Python, using the TensorFlow library. Preliminary results demonstrate effectiveness in classifying commands, with classification accuracies for ten trained models showing promise. Successful classification was achieved with models showing accuracy in the range of 66.6% to 100%. An average word classification accuracy of 82.5% between all models is observed. While all models were trained and tested on the same datasets, the stochastic nature of the model has significant effects on output, with the dropout layer adding artificial noise to training, and the gradient-descent based optimization algorithm adding random variance to the completed model effectiveness.