Prototypical Metric Transfer Learning for Continuous Speech Keyword Spotting with Limited Training Data

  • Harshita Seth
  • Pulkit Kumar
  • Muktabh Mayank SrivastavaEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 950)


Continuous Speech Keyword Spotting (CSKS) is the problem of spotting keywords in recorded conversations, when a small number of instances of keywords are available in training data. Unlike the more common Keyword Spotting, where an algorithm needs to detect lone keywords or short phrases like “Alexa”, “Cortana”, “Hi Alexa!”, “Whatsup Octavia?” etc. in speech, CSKS needs to filter out embedded words from a continuous flow of speech, ie. spot “Anna” and “github” in “I know a developer named Anna who can look into this github issue.” Apart from the issue of limited training data availability, CSKS is an extremely imbalanced classification problem. We address the limitations of simple keyword spotting baselines for both aforementioned challenges by using a novel combination of loss functions (Prototypical networks’ loss and metric loss) and transfer learning. Our method improves F1 score by over 10%.


Audio keyword detection Prototypical metric loss Few-shot Transfer learning 


  1. 1.
    Sainath, T.N., Parada, C.: Convolutional neural networks for small-footprint keyword spotting. In: Interspeech (2015)Google Scholar
  2. 2.
    Tang, R., Lin, J.: Honk: a PyTorch reimplementation of convolutional neural networks for keyword spotting. arXiv preprint arXiv:1710.06554, 18 October 2017
  3. 3.
    Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., Ng, A.Y.: Deep speech: scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567, 17 December 2014
  4. 4.
    Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, pp. 4077–4087 (2017)Google Scholar
  5. 5.
    Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition, vol. 12, pp. 84–92. Springer, Cham (2015)CrossRefGoogle Scholar
  6. 6.
    Weintraub, M.: Keyword-spotting using SRI’s DECIPHER large-vocabulary speech-recognition system. In: 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing 1993, ICASSP 1993, vol. 2, pp. 463–466. IEEE, 27 April 1993Google Scholar
  7. 7.
    Wilpon, J.G., Rabiner, L.R., Lee, C.H., Goldman, E.R.: Automatic recognition of keywords in unconstrained speech using hidden Markov models. IEEE Transact. Acoust. Speech Sig. Proc. 38(11), 1870–1878 (1990)CrossRefGoogle Scholar
  8. 8.
    Rose, R.C., Paul, D.B.: A hidden Markov model based keyword recognition system. In: 1990 International Conference on Acoustics, Speech, and Signal Processing 1990, ICASSP 1990, pp. 129–132. IEEE, 3 April 1990Google Scholar
  9. 9.
    Nouza, J., Silovsky, J.: Fast keyword spotting in telephone speech. Radioengineering 18(4), 665–70 (2009)Google Scholar
  10. 10.
    Lee, H., Pham, P., Largman, Y., Ng, A.Y.: Unsupervised feature learning for audio classification using convolutional deep belief networks. In: Advances in Neural Information Processing Systems, pp. 1096–1104 (2009)Google Scholar
  11. 11.
    Mohamed, A.R., Dahl, G.E., Hinton, G.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20(1), 14–22 (2012)CrossRefGoogle Scholar
  12. 12.
    Grosse, R., Raina, R., Kwong, H., Ng, A.Y.: Shift-invariance sparse coding for audio classification. arXiv preprint arXiv:1206.5241, 20 June 2012
  13. 13.
    Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Kingsbury, B.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)CrossRefGoogle Scholar
  14. 14.
    Dahl, G.E., Yu, D., Deng, L., Acero, A.: Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transact. Audio Speech Lang. Process. 20(1), 30–42 (2012)CrossRefGoogle Scholar
  15. 15.
    Li, K.P., Naylor, J.A., Rossen, M.L.: A whole word recurrent neural network for keyword spotting. In: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing 1992, ICASSP 1992, vol. 2, pp. 81–84. IEEE, 23 March 1992Google Scholar
  16. 16.
    Fernández, S., Graves, A., Schmidhuber, J.: An application of recurrent neural networks to discriminative keyword spotting. In: International Conference on Artificial Neural Networks, vol. 9, pp. 220–229. Springer, Heidelberg, September 2007Google Scholar
  17. 17.
    Chen, G., Parada, C., Heigold, G.: Small-footprint keyword spotting using deep neural networks. In: ICASSP, vol. 14, pp. 4087–4091, 4 May 2014Google Scholar
  18. 18.
    Pons, J., Serrá, J., Serra, X.: Training neural audio classifiers with few data. arXiv preprint arXiv:1810.10274, 24 October 2018
  19. 19.
    Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D.: Matching networks for one shot learning. In: Advances in Neural Information Processing Systems, pp. 3630–3638 (2016)Google Scholar
  20. 20.
    Kunze, J., Kirsch, L., Kurenkov, I., Krug, A., Johannsmeier, J., Stober, S.: Transfer learning for speech recognition on a budget. arXiv preprint arXiv:1706.00290, 1 June 2017
  21. 21.
    Choi, K., Fazekas, G., Sandler, M., Cho, K.: Transfer learning for music classification and regression tasks. arXiv preprint arXiv:1703.09179, 27 March 2017
  22. 22.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  23. 23.
    Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting.
  24. 24.
    Rohlicek, J.R., Russell, W., Roukos, S., Gish, H.: Continuous hidden Markov modeling for speaker-independent word spotting. In: 1989 International Conference on Acoustics, Speech, and Signal Processing 1989, ICASSP 1989, pp. 627–630. IEEE, 23 May 1989Google Scholar
  25. 25.
    Wilpon, J.G., Miller, L.G., Modi, P.: Improvements and applications for key word recognition using hidden Markov modeling techniques. In: 1991 International Conference on Acoustics, Speech, and Signal Processing 1991, ICASSP 1991, pp. 309–312. IEEE, 14 April 1991Google Scholar
  26. 26.
    Silaghi, M.C., Bourlard, H.: Iterative posterior-based keyword spotting without filler models. In: Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, pp. 213–216, 12 December 1999Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Harshita Seth
    • 1
  • Pulkit Kumar
    • 1
  • Muktabh Mayank Srivastava
    • 1
    Email author
  1. 1.Paralleldots, Inc.GurgaonIndia

Personalised recommendations