Abstract
Intent Recognition (IR) is considered a key area in Natural Language Processing (NLP). It has crucial usage in various applications. One is the Search Engine- Interpreting the context of text searched by the user improves the response time and helps the search engines give appropriate outputs. Another can be Social Media Analytics-Analysing profiles of users on different social media platforms has become a necessity in today’s applications like recommendation systems in the online world, digital marketing, and a lot more. Many researchers are using different techniques for achieving intent recognition but getting high accuracy in intent recognition is crucial. In this work, named BERT-IR, a pre-trained Natural Language Processing model called as BERT model, along with few add-ons, is applied for the task of Intent Recognition. We have achieved an accuracy of 97.67% on a widely used dataset which shows the capability and efficiency of our work. For comparison purposes, we have applied primarily used Machine Learning techniques, namely Naive Bayes, Logistic Regression, Decision Tree, Random Forest, and Gradient Boost as well as Deep Learning Techniques used for intent recognition like Recurrent Neural Network, Long Short Term Memory Network, and Bidirectional Long Short Term Memory Network on the same dataset and evaluated the accuracy. It is found out that BERT-IR’s accuracy is far better than that of the other models implemented.
Article PDF
Avoid common mistakes on your manuscript.
References
A.S. Ahmad, M.Y. Hassan, M.P. Abdullah, H.A. Rahman, F. Hussin, H. Abdullah, et al., A review on applications of ANN and SVM for building electrical energy consumption forecasting, Renewable and Sustainable Energy Reviews 33 (2014), 102–109.
S. Bao, H. He, F. Wang, H. Wu, H. Wang, PLATO: Pre-trained dialogue generation model with discrete latent variable, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 85–96.
Y. Bengio, R. Ducharme, P. Vincent, C. Janvin, A neural probabilistic language model, The journal of machine learning research 3 (2003), 1137–1155.
A. Bhargava, A. Celikyilmaz, D. Hakkani-Tür, R. Sarikaya, Easy contextual intent prediction and slot detection, 2013 IEEE international conference on acoustics, speech and signal processing, IEEE, Vancouver, BC, Canada, 2013, pp. 8337–8341.
A. Celikyilmaz, D. Hakkani-Tur, G. Tur, A. Fidler, D. Hillard, Exploiting distance based similarity in topic models for user intent detection, 2011 IEEE Workshop on Automatic Speech Recognition & Understanding, IEEE, Waikoloa, HI, USA, 2011, pp. 425–430.
Y.N. Dauphin, G. Tur, D. Hakkani-Tur, L. Heck, Zero-shot learning for semantic utterance classification, arXiv preprint arXiv:1401.0509, 2013.
J. Devlin, M.W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
R. Dey, F.M. Salem, Gate-variants of gated recurrent unit (GRU) neural networks, 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), IEEE, Boston, MA, USA, 2017, pp. 1597–1600.
A. Genkin, D.D. Lewis, D. Madigan, Large-scale bayesian logistic regression for text categorization, technometrics 49 (2007), 291–304.
P. Haffner, G. Tur, J.H. Wright, Optimizing SVMs for complex call classification, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., IEEE, Hong Kong, China, 2003, pp. I–I.
H.B. Hashemi, A. Asiaee, R. Kraft, Query intent detection using convolutional neural networks, International Conference on Web Search and Data Mining, Workshop on Query Understanding, ACM, 2016, pp. 1–5.
S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997), 1735–1780.
J.K. Kim, G. Tur, A. Celikyilmaz, B. Cao, Y.Y. Wang, Intent detection using semantically enriched word embeddings, 2016 IEEE Spoken Language Technology Workshop (SLT), IEEE, San Diego, CA, USA, 2016, pp. 414–419.
Y. Kim, Convolutional neural networks for sentence classification, arXiv, 2014.
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (1998), 2278–2324.
Z. Lin, M. Feng, C.N. dos Santos, M. Yu, B. Xiang, B. Zhou, et al., A structured self-attentive sentence embedding, 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. OpenReview.net. Available from: https://openreview.net/forum?id=BJC_jUqxe.
J. Liu, Y. Li, M. Lin, Review of intent detection methods in the human–machine dialogue system, Journal of Physics: Conference Series, IOP Publishing, 1267 (2019), 012059.
Q. Liu, J. Wang, D. Zhang, Y. Yang, N. Wang, Text features extraction based on TF-IDF associating semantic, 2018, pp. 2338–2343.
T.L. Luong, M.S. Cao, D.T. Le, X.H. Phan, Intent extraction from social media texts using sequential segmentation and deep learning models, 2017 9th International Conference on Knowledge and Systems Engineering (KSE), IEEE, Hue, Vietnam, 2017, pp. 215–220.
A. McCallum, K. Nigam, A comparison of event models for naive bayes text classification, AAAI-98 workshop on learning for text categorization, AAAI Press, Madison, Wisconsin, 1998, pp. 41–48.
M.E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 2227–2237.
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, Improving language understanding with unsupervised learning, Technical report, OpenAI, 2018.
H. Rashkin, E.M. Smith, M. Li, Y.L. Boureau, Towards empathetic open-domain conversation models: a new benchmark and dataset, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 5370–5381.
S. Ravuri, A. Stolcke, Recurrent neural network and LSTM models for lexical utterance classification, Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany, 2015, pp. 135–139.
S. Ravuri, A. Stolcke, A comparative study of recurrent neural network models for lexical domain classification, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, Shanghai, China, 2016, pp. 6075–6079.
F. Ren, Y. Bao, A review on human-computer interaction and intelligent robots, International Journal of Information Technology & Decision Making 19 (2020), 5–47.
R.E. Schapire, Y. Singer, BoosTexter: a boosting-based system for text categorization, Machine learning 39 (2000), 135–168.
W.L. Taylor, “cloze procedure”: a new tool for measuring readability, Journalism quarterly 30 (1953), 415–433.
O.T. Tran, T.C. Luong, Understanding what the users say in chatbots: a case study for the Vietnamese language, Engineering Applications of Artificial Intelligence 87 (2020), 103322.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, et al., Attention is all you need, Advances in neural information processing systems, Curran Associates, Inc., Long Beach, CA, USA, 2017, pp. 5998–6008.
T. Wolf, V. Sanh, J. Chaumond, C. Delangue, Transfertransfo: a transfer learning approach for neural network based conversational agents, arXiv preprint arXiv:1901.08149, 2019.
Y. Wu, M. Schuster, Z. Chen, Q.V. Le, M. Norouzi, W. Macherey, et al., Google’s neural machine translation system: bridging the gap between human and machine translation, arXiv preprint arXiv:1609.08144, 2016.
C. Zhang, Y. Li, N. Du, W. Fan, P. Yu, Joint slot filling and intent detection via capsule neural networks, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, 5259–5267.
Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, et al., Aligning books and movies: towards story-like visual explanations by watching movies and reading books, 2015 IEEE International Conference on Computer Vision (ICCV), IEEE, Santiago, Chile, 2015, pp. 19–27.
Author information
Authors and Affiliations
Corresponding author
Additional information
Peer review under responsibility of KEO (Henan) Education Technology Co. Ltd
Rights and permissions
This is an open access article distributed under the CC BY-NC 4.0 license (https://doi.org/creativecommons.org/licenses/by-nc/4.0/).
About this article
Cite this article
Khan, V., Meenai, T.A. Pretrained Natural Language Processing Model for Intent Recognition (BERT-IR). Hum-Cent Intell Syst 1, 66–74 (2021). https://doi.org/10.2991/hcis.k.211109.001
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.2991/hcis.k.211109.001