Predicting Question Quality Using Recurrent Neural Networks

  • Stefan Ruseti
  • Mihai DascaluEmail author
  • Amy M. Johnson
  • Renu Balyan
  • Kristopher J. Kopp
  • Danielle S. McNamara
  • Scott A. Crossley
  • Stefan Trausan-Matu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10947)


This study assesses the extent to which machine learning techniques can be used to predict question quality. An algorithm based on textual complexity indices was previously developed to assess question quality to provide feedback on questions generated by students within iSTART (an intelligent tutoring system that teaches reading strategies). In this study, 4,575 questions were coded by human raters based on their corresponding depth, classifying questions into four categories: 1-very shallow to 4-very deep. Here we propose a novel approach to assessing question quality within this dataset based on Recurrent Neural Networks (RNNs) and word embeddings. The experiments evaluated multiple RNN architectures using GRU, BiGRU and LSTM cell types of different sizes, and different word embeddings (i.e., FastText and Glove). The most precise model achieved a classification accuracy of 81.22%, which surpasses the previous prediction results using lexical sophistication complexity indices (accuracy = 41.6%). These results are promising and have implications for the future development of automated assessment tools within computer-based learning environments.


Question asking Recurrent neural network Word embeddings 



This research was partially supported by the 644187 EC H2020 Realising an Applied Gaming Eco-system (RAGE) project, the FP7 2008-212578 LTfLL project, the Department of Education, Institute of Education Sciences - Grant R305A130124, as well as the Department of Defense, Office of Naval Research - Grants N00014140343 and N000141712300.


  1. 1.
    Snow, C.: Reading for Understanding Toward an R&D Program in Reading Comprehension. Rand Corporation, Santa Monica (2002)Google Scholar
  2. 2.
    Palincsar, A.S., Brown, A.L.: Interactive promote learning teaching independent from text to. Read. Teach. 39, 771–777 (1986)Google Scholar
  3. 3.
    Rosenshine, B., Meister, C.: Reciprocal teaching: a review of the research. Rev. Educ. Res. 64, 479–530 (1994)CrossRefGoogle Scholar
  4. 4.
    Rosenshine, B., Meister, C., Chapman, S.: Teaching students to generate questions: a review of the intervention studies. Rev. Educ. Res. 66, 181–221 (1996)CrossRefGoogle Scholar
  5. 5.
    McNamara, D.S., O’Reilly, T., Rowe, M., Boonthum, C., Levinstein, I.: iSTART: a web-based tutor that teaches self-explanation and metacognitive reading strategies. In: Reading Comprehension Strategies: Theories, Interventions, and Technologies, pp. 397–420 (2007)Google Scholar
  6. 6.
    VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rosé, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 31, 3–62 (2007)CrossRefGoogle Scholar
  7. 7.
    Graesser, A.C., McMahen, C.L.: Anomalous information triggers questions when adults solve quantitative problems and comprehend stories. J. Educ. Psychol. 85, 136–151 (1993)CrossRefGoogle Scholar
  8. 8.
    Wisher, R.A., Graesser, A.C.: Question-asking in advanced distributed learning environments. In: Toward a Science of Distributed Learning and Training, pp. 209–234. American Psychological Association, Washington, D.C. (2007)Google Scholar
  9. 9.
    Beck, I., McKeown, M.G., Hamilton, R.L., Kucan, L.: Questioning the Author: An Approach for Enhancing Student Engagement (1997).
  10. 10.
    Kintsch, W.: Comprehension: A Paradigm for Cognition. Cambridge University Press, New York (1998)Google Scholar
  11. 11.
    Graesser, A.C., Person, N.K.: Question asking during tutoring. Am. Educ. Res. J. 31, 104–137 (1994)CrossRefGoogle Scholar
  12. 12.
    Davey, B., McBride, S.: Effects of question-generation training on reading comprehension. J. Educ. Psychol. 78, 256–262 (1986)CrossRefGoogle Scholar
  13. 13.
    Craig, S.D., Gholson, B., Ventura, M., Graesser, A.C.: Overhearing dialogues and monologues in virtual tutoring sessions: effects on questioning and vicarious learning. Int. J. Artif. Intell. Educ. 11, 242–253 (2000)Google Scholar
  14. 14.
    Silva, J., Coheur, L., Mendes, A.C., Wichert, A.: From symbolic to sub-symbolic information in question classification. Artif. Intell. Rev. 35, 137–154 (2011)CrossRefGoogle Scholar
  15. 15.
    Ittycheriah, A., Franz, M., Zhu, W., Ratnaparkhi, A., Mammone, R.J.: IBM’s statistical question answering system. In: Proceedings of TREC-9 Conference, pp. 229–234 (2000)Google Scholar
  16. 16.
    Hovy, E., Gerber, L., Hermjakob, U., Lin, C.-Y., Ravichandran, D.: Toward semantics-based answer pinpointing. In: Proceedings of the First International Conference on Human Language Technology Research, pp. 1–7 (2001)Google Scholar
  17. 17.
    Harabagiu, S., Moldovan, D., Pasca, M., Mihalcea, R., Surdeanu, M., Bunescu, R., Girju, R., Rus, V., Morarescu, P.: FALCON: boosting knowledge for answer engines. In: Proceedings of Ninth Text Retrieval Conference (TREC 2000), pp. 479–488 (2000)Google Scholar
  18. 18.
    Gerber, L.: A QA Typology for Webclopedia (2001)Google Scholar
  19. 19.
    Li, X., Roth, D.: Learning question classifiers. In: Proceedings of the 19th International Conference on Computational Linguistics, vol. 1, pp. 1–7 (2002)Google Scholar
  20. 20.
    Bloom, B.S.: Taxonomy of Educational Objectives, Cognitive Domain, pp. 20–24. McKay, New York (1956)Google Scholar
  21. 21.
    Mosenthal, P.B.: Understanding the strategies of document literacy and their conditions of use. J. Educ. Psychol. 88, 314–332 (1996)CrossRefGoogle Scholar
  22. 22.
    Olney, A., Louwerse, M., Matthews, E., Marineau, J., Hite-Mitchell, H., Graesser, A.: Utterance classification in AutoTutor. In: Proceedings of HLT-NAACL 2003 Workshop on Building Educational Applications Using Natural Language Processing, vol. 2, pp. 1–8. ACL, Morristown (2003)Google Scholar
  23. 23.
    Pietra, S.D., Pietra, V.D., Lafferty, J.: Inducing features of random fields. IEEE Trans. Pattern Anal. Mach. Intell. 19, 380–393 (1997)CrossRefGoogle Scholar
  24. 24.
    Chinchor, N., Robinson, P.: MUC-7 named entity task definition. In: Proceedings of the 7th Conference on Message Understanding, MUC6, p. 21 (1997)Google Scholar
  25. 25.
    Hacioglu, K., Ward, W.: Question classification with support vector machines and error correcting codes. In: Proceedings of HLT-NAACL 2003, pp. 28–30. Association for Computational Linguistics, Morristown (2003)Google Scholar
  26. 26.
    Zhang, D., Lee, W.S.: Question classification using support vector machines. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2003, p. 26. ACM Press, New York (2003)Google Scholar
  27. 27.
    Blunsom, P., Kocik, K., Curran, J.R.: Question classification with log-linear models. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2006, p. 615 (2006)Google Scholar
  28. 28.
    Kopp, K.J., Johnson, A.M., Crossley, S.A., McNamara, D.S.: Assessing question quality using NLP. In: André, E., Baker, R., Hu, X., Rodrigo, Ma.M.T., du Boulay, B. (eds.) AIED 2017. LNCS, vol. 10331, pp. 523–527. Springer, Cham (2017). Scholar
  29. 29.
    Krishnan, V., Das, S., Chakrabarti, S.: Enhanced answer type inference from questions using sequential models. In: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, HLT 2005, pp. 315–322 (2005)Google Scholar
  30. 30.
    Suzuki, J., Taira, H., Sasaki, Y., Maeda, E.: Question classification using HDAG kernel. In: Proceedings of the ACL 2003 Workshop on Multilingual Summarization and Question Answering, vol. 12, pp. 61–68 (2003)Google Scholar
  31. 31.
    Hermjakob, U.: Parsing and question classification for question answering. In: Proceedings of Working Open-domain Question Answering, vol. 12, pp. 1–6 (2001)Google Scholar
  32. 32.
    Mishra, M., Mishra, V.K., Sharma, H.R.: Question classification using semantic, syntactic and lexical features. Int. J. Web Semant. Technol. 4, 39–47 (2013)CrossRefGoogle Scholar
  33. 33.
    Elman, J.L.: Finding structure in time. Cogn. Sci. 14, 179–211 (1990)CrossRefGoogle Scholar
  34. 34.
    Yih, W., He, X., Meek, C.: Semantic parsing for single-relation question answering. In: Association for Computational Linguistics, pp. 643–648 (2014)Google Scholar
  35. 35.
    Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics, pp. 655–665 (2014)Google Scholar
  36. 36.
    Iyyer, M., Boyd-graber, J., Claudino, L., Socher, R., Daum, H.: A neural network for factoid question answering over paragraphs. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014)Google Scholar
  37. 37.
    Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (Almost) from scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)zbMATHGoogle Scholar
  38. 38.
    Fei, T., Heng, W.J., Toh, K.C., Qi, T.: Question classification for e-learning by artificial neural network. In: ICICS-PCM 2003, pp. 1757–1761. IEEE (2003)Google Scholar
  39. 39.
    Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of 2014 Conference on Empirical Methods Natural Language Processing (EMNLP 2014), pp. 1746–1751 (2014)Google Scholar
  40. 40.
    Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2267–2273 (2015)Google Scholar
  41. 41.
    Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical turk as a tool for experimental behavioral research. PLoS One 8, e57410 (2013)CrossRefGoogle Scholar
  42. 42.
    Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)Google Scholar
  43. 43.
    Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching Word Vectors with Subword Information (2016)Google Scholar
  44. 44.
    Mikolov, T., Corrado, G., Chen, K., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR 2013, pp. 1–12 (2013)Google Scholar
  45. 45.
    Darío Gutiérrez, E., Levy, R., Bergen, B.K.: Finding non-arbitrary form-meaning systematicity using string-metric learning for Kernel regression. In: ACL, pp. 2379–2388 (2016)Google Scholar
  46. 46.
    Hochreiter, S., Urgen Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)CrossRefGoogle Scholar
  47. 47.
    Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP 2014, pp. 1724–1734 (2014)Google Scholar
  48. 48.
    Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw. 18, 602–610 (2005)CrossRefGoogle Scholar
  49. 49.
    dos Santos, C., Tan, M., Xiang, B., Zhou, B.: Attentive Pooling Networks. CoRR, abs/1602.03609. 2, 4 (2016)Google Scholar
  50. 50.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations (2015)Google Scholar
  51. 51.
    Xiong, C., Zhong, V., Socher, R.: Dynamic coattention networks for question answering. In: ICLR Submission, pp. 1–13 (2017)Google Scholar
  52. 52.
    Seo, M., Kembhavi, A., Farhadi, A., Hajishirzi, H.: Bidirectional attention flow for machine comprehension. In: ICLR 2017 (2017)Google Scholar
  53. 53.
    Masala, M., Ruseti, S., Rebedea, T.: Sentence selection with neural networks using string kernels. Proc. Comput. Sci. 112, 1774–1782 (2017)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Stefan Ruseti
    • 1
  • Mihai Dascalu
    • 1
    • 2
    Email author
  • Amy M. Johnson
    • 3
  • Renu Balyan
    • 3
  • Kristopher J. Kopp
    • 3
  • Danielle S. McNamara
    • 3
  • Scott A. Crossley
    • 4
  • Stefan Trausan-Matu
    • 1
    • 2
  1. 1.Faculty of Automatic Control and ComputersUniversity “Politehnica” of BucharestBucharestRomania
  2. 2.Academy of Romanian ScientistsBucharestRomania
  3. 3.Institute for the Science of Teaching and LearningArizona State UniversityTempeUSA
  4. 4.Department of Applied Linguistics/ESLGeorgia State UniversityAtlantaUSA

Personalised recommendations