Abstract
Block-wise missingness in multimodal data poses a challenging barrier for the analysis over it, which is quite common in practical scenarios such as the multimedia intelligent tutoring systems (ITSs). In this work, we collected data from 194 undergraduates via a biology ITS which involves three modalities: student-system logfiles, facial expressions, and eye tracking. However, only 32 out of the 194 students had all three modalities and 83% of them were missing the facial expression data, eye tracking data, or both. To handle such a block-wise missing problem, we propose a Progressively Refined Imputation for Multi-modalities by auto-Encoder (PRIME), which trains the model based on single, pairwise, and entire modalities for imputation in a progressive manner, and therefore enables us to maximally utilize all the available data. We have evaluated PRIME against single-modality log-only (without missingness handling) and five state-of-the-art missing data handling methods on one important yet challenging student modeling task: to predict students’ learning gains. Our results show that using multimodal data as a result of missing data handling yields better prediction performance than using logfiles only, and PRIME outperforms other baseline methods for both learning gain prediction and data reconstruction tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
iMotions: www.imotions.com.
References
Azevedo, R., Taub, M., Mudrick, et al.: Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies. In: Handbook of Self-Regulation of Learning and Performance, vol. 2 (2018)
Azevedo, R., Mudrick, N.V., Taub, M., Bradbury, A.E.: 23 self-regulation in computer-assisted learning systems (2019)
Azevedo, R., et al.: Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: issues and challenges (2019)
Beaulieu-Jones, B.K., Greene, C.S., et al.: Semi-supervised learning of the electronic health record for phenotype stratification. JBI 64, 168–178 (2016)
Beaulieu-Jones, B.K., Moore, J.H.: Missing data imputation in the electronic health record using deeply learned autoencoders. In: Pacific Symposium on Biocomputing 2017, pp. 207–218 (2017)
Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring learning from gaze data during interaction with an environment to support self-regulated learning. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 229–238. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_24
Bosch, N.: Multimodal affect detection in the wild: accuracy, availability, and generalizability. In: Proceedings of the 2015 ICMI, pp. 645–649. ACM (2015)
Grafsgaard, J.F., et al.: The additive value of multimodal features for predicting engagement, frustration, and learning during tutoring. In: Proceedings of the 16th International Conference on Multimodal Interaction, pp. 42–49. ACM (2014)
Hastie, T., Tibshirani, R., Sherlock, G., Eisen, M., Brown, P., Botstein, D.: Imputing missing data for gene expression arrays (1999)
Jaques, N., Taylor, S., Sano, A., Picard, R.: Multimodal autoencoder: a deep learning approach to filling in missing sensor data and enabling better mood prediction. In: 2017 7th ACII, pp. 202–208. IEEE (2017)
Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: 13th ACM Multimedia, pp. 677–682. ACM (2005)
Mao, Y., Lin, C., Chi, M.: Deep learning vs. Bayesian knowledge tracing: student models for interventions. J. EDM 10(2), 28–54 (2018)
Shang, C., et al.: VIGAN: missing view imputation with generative adversarial networks. In: 2017 IEEE Big Data, pp. 766–775. IEEE (2017)
Taub, M., Azevedo, R.: How does prior knowledge influence eye fixations and sequences of cognitive and metacognitive SRL processes during learning with an intelligent tutoring system? Int. J. AIED 29(1), 1–28 (2019)
Taub, M., Mudrick, N.V., Azevedo, R., Millar, G.C., Rowe, J., Lester, J.: Using multi-channel data with multi-level modeling to assess in-game performance during gameplay with CRYSTAL ISLAND. Comput. Hum. Behav. 76, 641–655 (2017)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. Roy. Stat. Soc.: Ser. B (Methodol.) 58(1), 267–288 (1996)
Tran, L., et al.: Missing modalities imputation via cascaded residual autoencoder. In: Proceedings of the IEEE Conference on CVPR, pp. 1405–1414 (2017)
Troyanskaya, O., et al.: Missing value estimation methods for DNA microarrays. Bioinformatics 17(6), 520–525 (2001)
VanLehn, K., Lynch, C., et al.: The Andes physics tutoring system: lessons learned. Int. J. Artif. Intell. Educ. 15(3), 147–204 (2005)
Woolf, B.P.: Student modeling. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Advances in Intelligent Tutoring Systems. Studies in Computational Intelligence, vol. 308, pp. 267–279. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14363-2_13
Xiang, S., Yuan, L., et al.: Multi-source learning with block-wise missing data for Alzheimer’s disease prediction. In: 19th ACM SIGKDD, pp. 185–193. ACM (2013)
Yuan, L., et al.: Multi-source learning for joint analysis of incomplete multi-modality neuroimaging data. In: 18th ACM SIGKDD, pp. 1149–1157. ACM (2012)
Zhong, B., et al.: Emotion recognition with facial expressions and physiological signals. In: 2017 IEEE SSCI, pp. 1–8. IEEE (2017)
Acknowledgements
This research was supported by the NSF Grants: #1660878, #1726550, and #1651909.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, X., Kim, YJ., Taub, M., Azevedo, R., Chi, M. (2020). PRIME: Block-Wise Missingness Handling for Multi-modalities in Intelligent Tutoring Systems. In: Ro, Y., et al. MultiMedia Modeling. MMM 2020. Lecture Notes in Computer Science(), vol 11962. Springer, Cham. https://doi.org/10.1007/978-3-030-37734-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-37734-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37733-5
Online ISBN: 978-3-030-37734-2
eBook Packages: Computer ScienceComputer Science (R0)