Conceptualization of IMS that Estimates Learners’ Mental States from Learners’ Physiological Information Using Deep Neural Network Algorithm

  • Tatsunori MatsuiEmail author
  • Yoshimasa Tawatsuji
  • Siyuan Fang
  • Tatsuro Uno
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11528)


To improve the efficiency of teaching and learning, it is substantially important to know learners’ mental states during their learning processes. In this study, we tried to extract the relationships between the learner’s mental states and the learner’s physiological information complemented by the teacher’s speech acts using machine learning. The results of the system simulation showed that the system could estimate the learner’s mental states in high accuracy. Based on the construction of the system, we further discussed the concept of IMS and the necessary future work for IMS development.


Intelligent mentoring system Physiological information Deep neural network 


  1. 1.
    Takehana, K., Matsui, T.: Association rules on relationships between learner’s physiological information and mental states during learning process. In: Yamamoto, S. (ed.) HIMI 2016. LNCS, vol. 9735, pp. 209–219. Springer, Cham (2016). Scholar
  2. 2.
    Fujiyoshi, H., Yoshimura, K., Kunze, K., Kise, K.: Eibun mondai kaitoji no shiten joho o mochiita eigo noryoku suiteho [English ability estimation method using eye movement information during English question answering]. Tech. Rep. Inst. Electron. Inf. Commun. Eng. 115(25), 49–54 (2015)Google Scholar
  3. 3.
    Horiguchi, Y., Kojima, K., Matsui, T.: A method to estimate learners’ impasses based on features in low-level interactions by using MRA. In: Proceedings of the 58th SIG on Advanced Learning Science and Technology, pp. 1–6 (2010)Google Scholar
  4. 4.
    Kojima, K., Muramatsu, K., Matsui, T.: Experimental study on description of eye-movements among choices in answering to multiple-choice problems. Trans. Jpn. Soc. Inf. Syst. Educ. 31(2), 197–202 (2014)Google Scholar
  5. 5.
    D’Mello, S., Graesser, A., Picard, R.W.: Toward an affect-sensitive AutoTutor. IEEE Intell. Educ. Syst. 22(4), 53–61 (2007)CrossRefGoogle Scholar
  6. 6.
    Matsui, T., Horiguchi, Y., Kojima, K., Akakura, T.: A study on exploration of relationships between behaviors and mental states of learners for value co-creative education and learning environment. In: Yamamoto, S. (ed.) HCI 2014. LNCS, vol. 8522, pp. 69–79. Springer, Cham (2014). Scholar
  7. 7.
    Ryu, H., Monk, A.: Analysing interaction problems with cyclic interaction theory: low-level interaction walkthrough. PsychNology J. 2(3), 304–330 (2004)Google Scholar
  8. 8.
    Fujie, Y.: Role of teacher’s repetition in classroom teaching. Jpn. J. Educ. Technol. 23(4), 201–212 (2000)Google Scholar
  9. 9.
    Kishi, T., Nojima, E.: A structural analysis of elementary school teachers’ and children’s utterances in Japanese classes. Jpn. J. Educ. Psychol. 54(3), 322–333 (2006)CrossRefGoogle Scholar
  10. 10.
    Shimizu, Y., Uchida, N.: How do children adapt to classroom discourse? Quantitative and qualitative analyses of first grade homeroom activities. Jpn. J. Educ. Psychol. 49(3), 314–325 (2001)CrossRefGoogle Scholar
  11. 11.
    Pekrun, R., Goetz, T., Frenzel, A.C., Barchfeld, P., Perry, R.P.: Measuring emotions in students’ learning and performance: the achievement emotions questionnaire (AEQ). Contemp. Educ. Psychol. 36(1), 36–48 (2011)CrossRefGoogle Scholar
  12. 12.
    Tawatsuji, Y., Uno, T., Fang, S., Matsui, T.: Real-time estimation of learners’ mental states from learners’ physiological information using deep learning, In: Yang, J.C., et al. (eds.) Proceedings of the 26th International Conference on Computers in Education, pp. 107–109 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Tatsunori Matsui
    • 1
    Email author
  • Yoshimasa Tawatsuji
    • 1
  • Siyuan Fang
    • 2
  • Tatsuro Uno
    • 3
  1. 1.Faculty of Human SciencesWaseda UniversityTokorozawaJapan
  2. 2.Global Education CenterWaseda UniversityShinjukuJapan
  3. 3.School of Human SciencesWaseda UniversityTokorozawaJapan

Personalised recommendations