Advertisement

Attention-Based CNN-BLSTM Networks for Joint Intent Detection and Slot Filling

  • Yufan Wang
  • Li Tang
  • Tingting He
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11221)

Abstract

Dialogue intent detection and semantic slot filling are two critical tasks in nature language understanding (NLU) for task-oriented dialog systems. In this paper, we present an attention-based encoder-decoder neural network model for joint intent detection and slot filling, which encodes sentence representation with a hybrid Convolutional Neural Networks and Bidirectional Long Short-Term Memory Networks (CNN-BLSTM), and decodes it with an attention-based recurrent neural network with aligned inputs. In the encoding process, our model firstly extracts higher-level phrase representations and local features from each utterance using convolutional neural network, and then propagates historical contextual semantic information with a bidirectional long short-term memory network layer architecture. Accordingly, we could obtain sentence representation by merging the two architectures mentioned above. In the decoding process, we introduce attention mechanism in long short-term memory networks that can provide additional sematic information. We conduct experiment on dialogue intent detection and slot filling tasks with standard data set Airline Travel Information System (ATIS). Experimental results manifest that our proposed model can achieve better overall performance.

Keywords

Nature language understanding Slot filling Intent detection Attention model 

Notes

Acknowledgments

This research is supported by the Fundamental Research Funds for the Central Universities (CCNU18JCK05), the National Natural Science Foundation of China (61532008), the National Science Foundation of China (61572223), the National Key Research and Development Program of China (2017YFC0909502).

References

  1. 1.
    Shen, B., Inkpen, D.: Speech intent recognition for robots (2017)Google Scholar
  2. 2.
    Liu, B., Lane, I.: Attention-based recurrent neural network models for joint intent detection and slot filling (2016)Google Scholar
  3. 3.
    Liu, B., Line, I.: Recurrent neural network structured output prediction for spoken language understanding (2015)Google Scholar
  4. 4.
    Kim, Y.: Convolutional neural networks for sentence classification. Eprint Arxiv (2014)Google Scholar
  5. 5.
    Zhang, X., Wang, H.: A joint model of intent determination and slot filling for spoken language understanding, pp. 5690–5694 (2016)Google Scholar
  6. 6.
    Haffner, P., Tur, G., Wright, J.H.: Optimizing SVMs for complex call classification (2003)Google Scholar
  7. 7.
    Chen, J., Huang, H., Tian, S., et al.: Feature selection for text classification with Naive Bayes. Expert Syst. Appl. Int. J. 36(3), 5432–5435 (2009)CrossRefGoogle Scholar
  8. 8.
    Graves, A., Jaitly, N., Mohamed, A.R.: Hybrid speech recognition with deep bidirectional LSTM (2014)Google Scholar
  9. 9.
    Xiao, Y., Cho, K.: Efficient character-level document classification by combining convolution and recurrent layers (2016)Google Scholar
  10. 10.
    Xiao, J., Wang, X., Liu, B.: The study of a nonstationary maximum entropy Markov model and its application on the pos-tagging task. ACM Trans. Asian Lang. Inf. Process. 6(2), 7 (2007)CrossRefGoogle Scholar
  11. 11.
    Raymond, C., Riccardi, G.: Generative and discriminative algorithms for spoken language understanding (2007)Google Scholar
  12. 12.
    Aliannejadi, M., Kiaeeha, M., Khadivi, S., et al.: Graph-based semi-supervised conditional random fields for spoken language understanding using unaligned data (2017)Google Scholar
  13. 13.
    Xu, P., Sarikaya, R.: Convolutional neural network based triangular CRF for joint intent detection and slot filling (2014)Google Scholar
  14. 14.
    Yao, K., Peng, B., Zhang, Y., et al.: Spoken language understanding using long short-term memory neural networks (2015)Google Scholar
  15. 15.
    Vu, N.T., Gupta, P., Adel, H., et al.: Bi-directional recurrent neural network with ranking loss for spoken language understanding (2016)Google Scholar
  16. 16.
    Kurata, G., Xiang, B., Zhou, B., et al.: Leveraging sentence-level information with encoder LSTM for natural language understanding (2016)Google Scholar
  17. 17.
    Zhu, S., Yu, K.: Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding (2017)Google Scholar
  18. 18.
    Guo, D., Tur, G., Yih, W.T., et al.: Joint semantic utterance classification and slot filling with recursive neural networks (2015)Google Scholar
  19. 19.
    Liu, B., Lane, I.: Joint online spoken language understanding and language modeling with recurrent neural networks (2016)Google Scholar
  20. 20.
    Weigelt, S., Hey, T., Landhäußer, M.: Integrating a dialog component into a framework for spoken language understanding (2018)Google Scholar
  21. 21.
    Zhou, C., Sun, C., Liu, Z., et al.: A C-LSTM neural network for text classification. Comput. Sci. 1(4), 39–44 (2015)Google Scholar
  22. 22.
    Yao, K., Peng, B., Zhang, Y., et al.: Spoken language understanding using long short-term memory neural networks. In: IEEE – Institute of Electrical & Electronics Engineers, pp. 189–194 (2014)Google Scholar
  23. 23.
  24. 24.
    Yin, W., Schütze, H., Xiang, B., et al.: ABCNN: attention-based convolutional neural network for modeling sentence pairs (2015)Google Scholar
  25. 25.
    Morin, F., Bengio, Y.: Hierarchical probabilistic neural network language model. Aistats (2005)Google Scholar
  26. 26.
    Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks (2013)Google Scholar
  27. 27.
    Hemphill, C.T., Godfrey, J.J., Doddington, G.R.: The ATIS spoken language systems pilot corpus. In: Proceedings of the Darpa Speech & Natural Language Workshop, pp. 96–101 (1990)Google Scholar
  28. 28.
    Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. In: International Conference on Machine Learning, pp. 2342–2350. JMLR.org (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of ComputerCentral China Normal UniversityWuhanChina

Personalised recommendations