Expert Feature-Engineering vs. Deep Neural Networks: Which Is Better for Sensor-Free Affect Detection?

  • Yang JiangEmail author
  • Nigel Bosch
  • Ryan S. Baker
  • Luc Paquette
  • Jaclyn Ocumpaugh
  • Juliana Ma. Alexandra L. Andres
  • Allison L. Moore
  • Gautam Biswas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10947)


The past few years have seen a surge of interest in deep neural networks. The wide application of deep learning in other domains such as image classification has driven considerable recent interest and efforts in applying these methods in educational domains. However, there is still limited research comparing the predictive power of the deep learning approach with the traditional feature engineering approach for common student modeling problems such as sensor-free affect detection. This paper aims to address this gap by presenting a thorough comparison of several deep neural network approaches with a traditional feature engineering approach in the context of affect and behavior modeling. We built detectors of student affective states and behaviors as middle school students learned science in an open-ended learning environment called Betty’s Brain, using both approaches. Overall, we observed a tradeoff where the feature engineering models were better when considering a single optimized threshold (for intervention), whereas the deep learning models were better when taking model confidence fully into account (for discovery with models analyses).


Student modeling Feature engineering Deep learning Deep neural networks Affect and behavior detection Betty’s brain 



We would like to thank the National Science Foundation (NSF) for their support (#DRL-1561567).


  1. 1.
    Clarke-Midura, J., Yudelson, M.V.: Towards identifying students’ causal reasoning using machine learning. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 704–707. Springer, Heidelberg (2013). Scholar
  2. 2.
    Rowe, E., Asbell-Clarke, J., Baker, R.S., Eagle, M., Hicks, A.G., Barnes, T.M., Brown, R.A., Edwards, T.: Assessing implicit science learning in digital games. Comput. Hum. Behav. 76, 617–630 (2017)CrossRefGoogle Scholar
  3. 3.
    Baker, R.S., Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., Salvi, A., Aleven, V., Kusbit, G.W., Ocumpaugh, J., Rossi, L.: Towards sensor-free affect detection in cognitive tutor algebra. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 126–133 (2012)Google Scholar
  4. 4.
    D’Mello, S., Jackson, T., Craig, S., Morgan, B., Chipman, P., White, H., Person, N., Kort, B., Kaliouby, R.e., Picard, R., Graesser, A.: AutoTutor detects and responds to learners affective and cognitive states. In: Proceedings of the Workshop on Emotional and Cognitive Issues in ITS in Conjunction with the 9th International Conference on ITS, pp. 31–43 (2008)Google Scholar
  5. 5.
    Pardos, Z.A., Baker, R.S., Pedro, M.O.C.Z.S., Gowda, S.M., Gowda, S.M.: Affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes. In: Proceedings of the 3rd International Conference on Learning Analytics and Knowledge, pp. 117–124 (2013)Google Scholar
  6. 6.
    Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. In: Proceedings of the 2009 Conference on Artificial Intelligence in Education (AIED 2009), pp. 17–24. IOS Press, Amsterdam (2009)Google Scholar
  7. 7.
    Baker, R.S., Corbett, A.T., Roll, I., Koedinger, K.R.: Developing a generalizable detector of when students game the system. User Model. User-Adapt. Interact. 18, 287–314 (2008)CrossRefGoogle Scholar
  8. 8.
    Cetintas, S., Si, L., Xin, Y.P., Hord, C.: Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques. IEEE Trans. Learn. Technol. 3, 228–236 (2010)CrossRefGoogle Scholar
  9. 9.
    Kai, S., Paquette, L., Baker, R.S., Bosch, N., D’Mello, S., Ocumpaugh, J., Shute, V., Ventura, M.: A comparison of video-based and interaction-based affect detectors in physics playground. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 77–84 (2015)Google Scholar
  10. 10.
    Ocumpaugh, J., Baker, R.S., Rodrigo, M.M.T.: Baker Rodrigo Ocumpaugh Monitoring Protocol (BROMP) 2.0 technical and training manual. Technical report, Teachers College, Columbia University, Ateneo Laboratory for the Learning Sciences (2015)Google Scholar
  11. 11.
    Fancsali, S.E.: Causal discovery with models: behavior, affect, and learning in cognitive tutor algebra. In: Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), pp. 28–35 (2014)Google Scholar
  12. 12.
    San Pedro, M.O.Z., Baker, R.S., Bowers, A.J., Heffernan, N.T.: Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. In: Proceedings of the 6th International Conference on Educational Data Mining, pp. 177–184 (2013)Google Scholar
  13. 13.
    San Pedro, M.O.Z., Snow, E.L., Baker, R.S., McNamara, D.S., Heffernan, N.T.: Exploring Dynamic Assessments of Affect, Behavior, and Cognition and Math State Test Achievement. In: Proceedings of the 8th International Conference on Educational Data Mining, pp. 85–92 (2015)Google Scholar
  14. 14.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar
  15. 15.
    Botelho, A.F., Baker, R.S., Heffernan, N.T.: Improving sensor-free affect detection using deep learning. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 40–51. Springer, Cham (2017). Scholar
  16. 16.
    Khajah, M., Lindsey, R.V., Mozer, M.C.: How deep is knowledge tracing? In: Proceedings of the 9th International Conference on Educational Data Mining (EDM 2016), pp. 94–101 (2016)Google Scholar
  17. 17.
    Lin, C., Chi, M.: A comparisons of BKT, RNN and LSTM for learning gain prediction. In: André, E., Baker, R., Hu, X., Rodrigo, M.M.T., du Boulay, B. (eds.) AIED 2017. LNCS (LNAI), vol. 10331, pp. 536–539. Springer, Cham (2017). Scholar
  18. 18.
    Piech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., Sohl-Dickstein, J.: Deep knowledge tracing. In: Advances in Neural Information Processing Systems (NIPS 2015), vol. 28, pp. 505–513. Curran Associates, Inc. (2015)Google Scholar
  19. 19.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, pp. 1097–1105, Lake Tahoe, Nevada (2012)Google Scholar
  20. 20.
    Botelho, A.F., Baker, R.S., Heffernan, N.T.: Developing and evaluating “deep” sensor-free detectors of student affect. (Manuscript in preparation)Google Scholar
  21. 21.
    Leelawong, K., Biswas, G.: Designing learning by teaching agents: the Betty’s brain system. Int J. Artif. Intell. Educ. 18, 181–208 (2008)Google Scholar
  22. 22.
    Baker, R.S., Ocumpaugh, J., Gowda, S.M., Kamarainen, A.M., Metcalf, S.J.: Extending log-based affect detection to a multi-user virtual environment for science. In: Proceedings of the 22nd Conference on User Modelling, Adaptation, and Personalization, pp. 290–300 (2014)Google Scholar
  23. 23.
    Ocumpaugh, J., Baker, R.S., Rodrigo, M.M., Salvi, A., Velsen, M.V., Aghababyan, A., Martin, T.: HART: the human affect recording tool. In: Proceedings of the 33rd Annual International Conference on the Design of Communication (SIGDOC 2015). ACM, New York (2015)Google Scholar
  24. 24.
    Mierswa, I., Scholz, M., Klinkenberg, R., Wurst, M., Euler, T.: Rapid prototyping for complex data mining tasks. In: Proceedings of KDD 2006, pp. 935–940 (2006)Google Scholar
  25. 25.
    Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by Exponential Linear Units (ELUs). In: ICLR 2016 (2016)Google Scholar
  26. 26.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Paquette, L., Baker, R.S., Pedro, M.A.S., Gobert, J.D., Rossi, L., Nakama, A., Kauffman-Rogoff, Z.: Sensor-free affect detection for a simulation-based science inquiry learning environment. In: Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pp. 1–10 (2014)Google Scholar
  28. 28.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112. Curran Associates, Inc. (2014)Google Scholar
  29. 29.
    Ocumpaugh, J., Baker, R.S., Gowda, S.M., Heffernan, N.T., Heffernan, C.: Population validity for educational data mining: a case study in affect detection. Br. J. Educ. Psychol. 45, 487–501 (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Yang Jiang
    • 1
    Email author
  • Nigel Bosch
    • 2
  • Ryan S. Baker
    • 3
  • Luc Paquette
    • 2
  • Jaclyn Ocumpaugh
    • 3
  • Juliana Ma. Alexandra L. Andres
    • 3
  • Allison L. Moore
    • 4
  • Gautam Biswas
    • 4
  1. 1.Teachers CollegeColumbia UniversityNew YorkUSA
  2. 2.University of Illinois at Urbana-ChampaignChampaignUSA
  3. 3.University of PennsylvaniaPhiladelphiaUSA
  4. 4.Vanderbilt UniversityNashvilleUSA

Personalised recommendations